[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:39:43 dist-test-slave-1wns krb5kdc[1227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:39:43 dist-test-slave-1wns krb5kdc[1227](info): set up 2 sockets
Sep 26 13:39:43 dist-test-slave-1wns krb5kdc[1227](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 13:39:44.945503 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:39:45 dist-test-slave-1wns krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:39:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:39:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:39:45.088369 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:36007
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:36007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:45.207144 1240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:45.207518 1240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:45.207598 1240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:45.211927 1240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:39:45.212057 1240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:45.212091 1240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:45.212124 1240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:39:45.212160 1240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:39:45.217846 1240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:36007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:36007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:45.219136 1240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:45.220317 1240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:45.230648 1245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:45.230659 1248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:45.230659 1246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:45.230710 1240 server_base.cc:1034] running on GCE node
I20260926 13:39:45.231534 1240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:45.232947 1240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:45.234138 1240 hybrid_clock.cc:648] HybridClock initialized: now 1790429985234048 us; error 107 us; skew 500 ppm
Sep 26 13:39:45 dist-test-slave-1wns krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:45.237306 1240 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:39:45.238523 1240 webserver.cc:460] Webserver started at http://127.1.49.62:36975/ using document root <none> and password file <none>
I20260926 13:39:45.239187 1240 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:45.239255 1240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:45.239445 1240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:45.241793 1240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "4d5f77fbe4634597ae9fd5a9ca295715"
format_stamp: "Formatted at 2026-09-26 13:39:45 on dist-test-slave-1wns"
server_key: "b706db1e84b6228d75343914f701c210"
server_key_iv: "19fc549a54c0d7fc8eda4fa87da86473"
server_key_version: "encryptionkey@0"
I20260926 13:39:45.242390 1240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "4d5f77fbe4634597ae9fd5a9ca295715"
format_stamp: "Formatted at 2026-09-26 13:39:45 on dist-test-slave-1wns"
server_key: "b706db1e84b6228d75343914f701c210"
server_key_iv: "19fc549a54c0d7fc8eda4fa87da86473"
server_key_version: "encryptionkey@0"
I20260926 13:39:45.246642 1240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 13:39:45.249689 1255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:45.251226 1240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:39:45.251463 1240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "4d5f77fbe4634597ae9fd5a9ca295715"
format_stamp: "Formatted at 2026-09-26 13:39:45 on dist-test-slave-1wns"
server_key: "b706db1e84b6228d75343914f701c210"
server_key_iv: "19fc549a54c0d7fc8eda4fa87da86473"
server_key_version: "encryptionkey@0"
I20260926 13:39:45.251588 1240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:45.282001 1240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:45.286162 1240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:45.286362 1240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:45.294411 1240 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:36007
I20260926 13:39:45.294569 1307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:36007 every 8 connection(s)
I20260926 13:39:45.296222 1240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:39:45.301219 1308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:45.306854 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 1240
I20260926 13:39:45.307009 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:39:45.308676 1220 external_mini_cluster.cc:1428] Setting key 9d2cf134ae9c08a75f1e133edd2be83a
I20260926 13:39:45.309294 1308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715: Bootstrap starting.
I20260926 13:39:45.312371 1308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:45.313314 1308 log.cc:824] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715: Log is configured to *not* fsync() on all Append() calls
I20260926 13:39:45.315562 1308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715: No bootstrap required, opened a new log
I20260926 13:39:45.320740 1308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5f77fbe4634597ae9fd5a9ca295715" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36007 } }
I20260926 13:39:45.321044 1308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:45.321112 1308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d5f77fbe4634597ae9fd5a9ca295715, State: Initialized, Role: FOLLOWER
I20260926 13:39:45.321650 1308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5f77fbe4634597ae9fd5a9ca295715" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36007 } }
I20260926 13:39:45.321818 1308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:39:45.321894 1308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:39:45.322003 1308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:45.323065 1308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5f77fbe4634597ae9fd5a9ca295715" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36007 } }
I20260926 13:39:45.323479 1308 leader_election.cc:302] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d5f77fbe4634597ae9fd5a9ca295715; no voters:
I20260926 13:39:45.323915 1308 leader_election.cc:288] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:39:45.323994 1313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:39:45.324234 1313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [term 1 LEADER]: Becoming Leader. State: Replica: 4d5f77fbe4634597ae9fd5a9ca295715, State: Running, Role: LEADER
I20260926 13:39:45.324617 1313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5f77fbe4634597ae9fd5a9ca295715" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36007 } }
I20260926 13:39:45.325436 1308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:39:45.327117 1314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d5f77fbe4634597ae9fd5a9ca295715. Latest consensus state: current_term: 1 leader_uuid: "4d5f77fbe4634597ae9fd5a9ca295715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5f77fbe4634597ae9fd5a9ca295715" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36007 } } }
I20260926 13:39:45.327159 1315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d5f77fbe4634597ae9fd5a9ca295715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5f77fbe4634597ae9fd5a9ca295715" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36007 } } }
I20260926 13:39:45.327286 1314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [sys.catalog]: This master's current role is: LEADER
I20260926 13:39:45.327323 1315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715 [sys.catalog]: This master's current role is: LEADER
I20260926 13:39:45.327718 1318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 13:39:45 dist-test-slave-1wns krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:45.331830 1318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:39:45.338750 1318 catalog_manager.cc:1347] Generated new cluster ID: d94ad6dfd9b24177ba6ceecc2d3acdd4
I20260926 13:39:45.338835 1318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:39:45.340852 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:45.312265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.1.49.62:36007)
0926 13:39:45.312826 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:45.312847 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:45.314467 (+ 1620us) server_negotiation.cc:407] Connection header received
0926 13:39:45.318222 (+ 3755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:45.318251 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:45.318743 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:45.319192 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:45.322206 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:45.323478 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:45.325357 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:45.325953 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:45.326167 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:45.333474 (+ 7307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:45.333482 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:45.333503 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:45.333546 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:45.336014 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:45.336886 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:45.336895 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:45.336903 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:45.337046 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:45.337535 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:45.337541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:45.337544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:45.338264 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:45.338520 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:45.339498 (+ 978us) server_negotiation.cc:299] Negotiation successful
0926 13:39:45.339877 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":168,"threads_started":1}
I20260926 13:39:45.356096 1318 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:39:45.357385 1318 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:39:45.366701 1318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d5f77fbe4634597ae9fd5a9ca295715: Generated new TSK 0
I20260926 13:39:45.367965 1318 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:39:45.415882 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36007
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:45.543876 1334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:45.544209 1334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:45.544260 1334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:45.549974 1334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:45.550083 1334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:45.550155 1334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:39:45.556797 1334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36007
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:45.558072 1334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:45.559276 1334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:45.570209 1339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:45.570338 1334 server_base.cc:1034] running on GCE node
W20260926 13:39:45.570326 1340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:45.570209 1342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:45.571159 1334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:45.572008 1334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:45.573189 1334 hybrid_clock.cc:648] HybridClock initialized: now 1790429985573174 us; error 31 us; skew 500 ppm
Sep 26 13:39:45 dist-test-slave-1wns krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:45.576445 1334 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:39:45.577540 1334 webserver.cc:460] Webserver started at http://127.1.49.1:35057/ using document root <none> and password file <none>
I20260926 13:39:45.578145 1334 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:45.578222 1334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:45.578416 1334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:45.580852 1334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "e0604ee7e7fa4054a97e58963d3db3f2"
format_stamp: "Formatted at 2026-09-26 13:39:45 on dist-test-slave-1wns"
server_key: "ee1b929942fd4d654973220d9e765206"
server_key_iv: "5e6acb7f8bcdf50c27f358d032074f25"
server_key_version: "encryptionkey@0"
I20260926 13:39:45.581463 1334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "e0604ee7e7fa4054a97e58963d3db3f2"
format_stamp: "Formatted at 2026-09-26 13:39:45 on dist-test-slave-1wns"
server_key: "ee1b929942fd4d654973220d9e765206"
server_key_iv: "5e6acb7f8bcdf50c27f358d032074f25"
server_key_version: "encryptionkey@0"
I20260926 13:39:45.585917 1334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:39:45.588987 1349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:45.590600 1334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:39:45.590790 1334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "e0604ee7e7fa4054a97e58963d3db3f2"
format_stamp: "Formatted at 2026-09-26 13:39:45 on dist-test-slave-1wns"
server_key: "ee1b929942fd4d654973220d9e765206"
server_key_iv: "5e6acb7f8bcdf50c27f358d032074f25"
server_key_version: "encryptionkey@0"
I20260926 13:39:45.590946 1334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:45.603258 1334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:45.607496 1334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:45.607757 1334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:45.608616 1334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:39:45.609664 1334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:39:45.609721 1334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:45.609777 1334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:39:45.609835 1334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:45.621280 1334 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:45343
I20260926 13:39:45.621327 1462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:45343 every 8 connection(s)
I20260926 13:39:45.622319 1334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:39:45.623507 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 1334
I20260926 13:39:45.623622 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:39:45.623953 1220 external_mini_cluster.cc:1428] Setting key c431b8b368d7674f63590827b45c782c
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
Sep 26 13:39:45 dist-test-slave-1wns krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:45.645565 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:45.624692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:60797 (local address 127.1.49.62:36007)
0926 13:39:45.624928 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:45.624936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:45.626358 (+ 1422us) server_negotiation.cc:407] Connection header received
0926 13:39:45.628474 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:45.628477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:45.628540 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:45.628623 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:45.630495 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:45.631201 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:45.633486 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:45.633804 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:45.633916 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:45.640326 (+ 6410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:45.640330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:45.640332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:45.640367 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:45.642258 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:45.643275 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:45.643279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:45.643280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:45.643381 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:45.644039 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:45.644043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:45.644044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:45.644285 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:45.644378 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:45.645281 (+ 903us) server_negotiation.cc:299] Negotiation successful
0926 13:39:45.645420 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 13:39:45.647094 1465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:45.625342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36007 (local address 127.1.49.1:60797)
0926 13:39:45.626174 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:45.626247 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:45.628210 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:45.628860 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:45.628878 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:45.629476 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:45.630272 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:45.630299 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:45.631399 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:45.631403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:45.633288 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:45.633305 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:45.634024 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:45.634030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:45.634286 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:45.635159 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:45.635182 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:45.640036 (+ 4854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:45.642454 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:45.642466 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:45.642524 (+ 58us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:45.643087 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:45.643546 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:45.643551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:45.643553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:45.643873 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:45.644419 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:45.644431 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:45.644916 (+ 485us) client_negotiation.cc:769] Sending connection context
0926 13:39:45.645200 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 13:39:45.645670 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":218,"threads_started":1}
I20260926 13:39:45.648904 1463 heartbeater.cc:347] Connected to a master server at 127.1.49.62:36007
I20260926 13:39:45.649390 1463 heartbeater.cc:464] Registering TS with master...
I20260926 13:39:45.650079 1463 heartbeater.cc:511] Master 127.1.49.62:36007 requested a full tablet report, sending...
I20260926 13:39:45.652130 1270 ts_manager.cc:194] Registered new tserver with Master: e0604ee7e7fa4054a97e58963d3db3f2 (127.1.49.1:45343)
I20260926 13:39:45.653400 1270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:60797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:39:45.661926 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36007
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:45.791481 1468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:45.791937 1468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:45.792023 1468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:45.797562 1468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:45.797674 1468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:45.797890 1468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:39:45.804636 1468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36007
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:45.805974 1468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:45.807188 1468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:45.820142 1473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:45.820302 1468 server_base.cc:1034] running on GCE node
W20260926 13:39:45.820173 1474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:45.820533 1476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:45.821091 1468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:45.821908 1468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:45.823132 1468 hybrid_clock.cc:648] HybridClock initialized: now 1790429985823107 us; error 40 us; skew 500 ppm
Sep 26 13:39:45 dist-test-slave-1wns krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:45.826980 1468 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:39:45.828172 1468 webserver.cc:460] Webserver started at http://127.1.49.2:42001/ using document root <none> and password file <none>
I20260926 13:39:45.828959 1468 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:45.829133 1468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:45.829391 1468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:45.831681 1468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "ecaa6cc245d54e63adb8701c0603bbf6"
format_stamp: "Formatted at 2026-09-26 13:39:45 on dist-test-slave-1wns"
server_key: "12354ef12a0c28e088a2396fb8349ec0"
server_key_iv: "25ebd48e891325be23f1b43a6047d367"
server_key_version: "encryptionkey@0"
I20260926 13:39:45.832412 1468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "ecaa6cc245d54e63adb8701c0603bbf6"
format_stamp: "Formatted at 2026-09-26 13:39:45 on dist-test-slave-1wns"
server_key: "12354ef12a0c28e088a2396fb8349ec0"
server_key_iv: "25ebd48e891325be23f1b43a6047d367"
server_key_version: "encryptionkey@0"
I20260926 13:39:45.837182 1468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:39:45.840384 1483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:45.842094 1468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:39:45.842279 1468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "ecaa6cc245d54e63adb8701c0603bbf6"
format_stamp: "Formatted at 2026-09-26 13:39:45 on dist-test-slave-1wns"
server_key: "12354ef12a0c28e088a2396fb8349ec0"
server_key_iv: "25ebd48e891325be23f1b43a6047d367"
server_key_version: "encryptionkey@0"
I20260926 13:39:45.842429 1468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:45.857199 1468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:45.861104 1468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:45.861338 1468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:45.862015 1468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:39:45.862963 1468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:39:45.863049 1468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:45.863229 1468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:39:45.863283 1468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:45.874675 1468 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:41427
I20260926 13:39:45.874691 1596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:41427 every 8 connection(s)
I20260926 13:39:45.875761 1468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:39:45.880141 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 1468
I20260926 13:39:45.880262 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:39:45.880563 1220 external_mini_cluster.cc:1428] Setting key 381f64db002602caa2881345921eb4ea
Sep 26 13:39:45 dist-test-slave-1wns krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
I20260926 13:39:45.893975 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:45.878025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:60867 (local address 127.1.49.62:36007)
0926 13:39:45.878211 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:45.878214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:45.879196 (+ 982us) server_negotiation.cc:407] Connection header received
0926 13:39:45.880918 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:45.880922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:45.880987 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:45.881094 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:45.882386 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:45.882995 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:45.884359 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:45.884651 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:45.884785 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:45.888662 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:45.888670 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:45.888674 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:45.888813 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:45.890872 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:45.891715 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:45.891719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:45.891721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:45.891867 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:45.892397 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:45.892400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:45.892402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:45.892651 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:45.892859 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:45.893683 (+ 824us) server_negotiation.cc:299] Negotiation successful
0926 13:39:45.893824 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
I20260926 13:39:45.895707 1599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:45.878391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36007 (local address 127.1.49.2:60867)
0926 13:39:45.878976 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:45.879025 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:45.880573 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:45.881301 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:45.881309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:45.881706 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:45.882234 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:45.882248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:45.883145 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:45.883156 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:45.884210 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:45.884220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:45.884873 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:45.884881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:45.885177 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:45.886037 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:45.886069 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:45.888404 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:45.891071 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:45.891079 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:45.891149 (+ 70us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:45.891503 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:45.892016 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:45.892019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:45.892020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:45.892227 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:45.892913 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:45.892925 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:45.893356 (+ 431us) client_negotiation.cc:769] Sending connection context
0926 13:39:45.893571 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 13:39:45.894070 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":126,"threads_started":1}
I20260926 13:39:45.897377 1597 heartbeater.cc:347] Connected to a master server at 127.1.49.62:36007
I20260926 13:39:45.897679 1597 heartbeater.cc:464] Registering TS with master...
I20260926 13:39:45.898310 1597 heartbeater.cc:511] Master 127.1.49.62:36007 requested a full tablet report, sending...
I20260926 13:39:45.899582 1270 ts_manager.cc:194] Registered new tserver with Master: ecaa6cc245d54e63adb8701c0603bbf6 (127.1.49.2:41427)
I20260926 13:39:45.900353 1270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:60867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:39:45.916122 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36007
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:46.041507 1602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:46.041843 1602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:46.041895 1602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:46.046716 1602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:46.046805 1602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:46.046903 1602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:39:46.052801 1602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36007
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:46.054119 1602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:46.055289 1602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:46.066727 1610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:46.066874 1602 server_base.cc:1034] running on GCE node
W20260926 13:39:46.066730 1607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:46.066730 1608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:46.067530 1602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:46.068394 1602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:46.069700 1602 hybrid_clock.cc:648] HybridClock initialized: now 1790429986069603 us; error 55 us; skew 500 ppm
Sep 26 13:39:46 dist-test-slave-1wns krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:46.073163 1602 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:39:46.074267 1602 webserver.cc:460] Webserver started at http://127.1.49.3:33881/ using document root <none> and password file <none>
I20260926 13:39:46.074958 1602 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:46.075029 1602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:46.075210 1602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:46.077603 1602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "22f8ceac43a14c0897e49a87a5fec285"
format_stamp: "Formatted at 2026-09-26 13:39:46 on dist-test-slave-1wns"
server_key: "94642c71bed49b327216377606901d5d"
server_key_iv: "626b69f53f30a938409feed94ea85775"
server_key_version: "encryptionkey@0"
I20260926 13:39:46.078200 1602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "22f8ceac43a14c0897e49a87a5fec285"
format_stamp: "Formatted at 2026-09-26 13:39:46 on dist-test-slave-1wns"
server_key: "94642c71bed49b327216377606901d5d"
server_key_iv: "626b69f53f30a938409feed94ea85775"
server_key_version: "encryptionkey@0"
I20260926 13:39:46.082441 1602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:39:46.085721 1617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:46.087219 1602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:39:46.087374 1602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "22f8ceac43a14c0897e49a87a5fec285"
format_stamp: "Formatted at 2026-09-26 13:39:46 on dist-test-slave-1wns"
server_key: "94642c71bed49b327216377606901d5d"
server_key_iv: "626b69f53f30a938409feed94ea85775"
server_key_version: "encryptionkey@0"
I20260926 13:39:46.087489 1602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:46.104437 1602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:46.108423 1602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:46.108618 1602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:46.109233 1602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:39:46.110153 1602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:39:46.110203 1602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:46.110248 1602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:39:46.110265 1602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:46.121748 1602 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:44533
I20260926 13:39:46.121778 1730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:44533 every 8 connection(s)
I20260926 13:39:46.122859 1602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:39:46.123965 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 1602
I20260926 13:39:46.124061 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:39:46.124346 1220 external_mini_cluster.cc:1428] Setting key be4e065b94feb118583c1d5c2cba3777
Sep 26 13:39:46 dist-test-slave-1wns krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:46.140559 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:46.125343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:44795 (local address 127.1.49.62:36007)
0926 13:39:46.125732 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:46.125736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:46.126582 (+ 846us) server_negotiation.cc:407] Connection header received
0926 13:39:46.128327 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:46.128330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:46.128382 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:46.128456 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:46.129871 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:46.130445 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:46.131965 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:46.132213 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:46.132335 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:46.135646 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:46.135650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:46.135653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:46.135689 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:46.137846 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:46.138606 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:46.138610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:46.138613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:46.138734 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:46.139202 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:46.139206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:46.139208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:46.139430 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:46.139554 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:46.140236 (+ 682us) server_negotiation.cc:299] Negotiation successful
0926 13:39:46.140384 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
I20260926 13:39:46.141585 1733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:46.125809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36007 (local address 127.1.49.3:44795)
0926 13:39:46.126369 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:46.126470 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:46.128096 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:46.128611 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:46.128619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:46.129129 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:46.129682 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:46.129697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:46.130607 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:46.130611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:46.131752 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:46.131764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:46.132419 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:46.132422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:46.132614 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:46.133310 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:46.133330 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:46.135297 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:46.138052 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:46.138060 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:46.138077 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:46.138455 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:46.138881 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:46.138885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:46.138887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:46.139077 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:46.139553 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:46.139563 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:46.139957 (+ 394us) client_negotiation.cc:769] Sending connection context
0926 13:39:46.140184 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 13:39:46.140580 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":146,"threads_started":1}
I20260926 13:39:46.143283 1731 heartbeater.cc:347] Connected to a master server at 127.1.49.62:36007
I20260926 13:39:46.143596 1731 heartbeater.cc:464] Registering TS with master...
I20260926 13:39:46.144304 1731 heartbeater.cc:511] Master 127.1.49.62:36007 requested a full tablet report, sending...
I20260926 13:39:46.145725 1270 ts_manager.cc:194] Registered new tserver with Master: 22f8ceac43a14c0897e49a87a5fec285 (127.1.49.3:44533)
I20260926 13:39:46.146368 1270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:44795
I20260926 13:39:46.152415 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:39:46 dist-test-slave-1wns krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429986, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:39:46 dist-test-slave-1wns krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429986, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.49.1@KRBTEST.COM
I20260926 13:39:46.184566 1738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:46.172045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.1.49.1:45343)
0926 13:39:46.172373 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:46.172379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:46.172404 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:39:46.172519 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:46.172524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:46.172725 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:46.172924 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:46.174214 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:46.175195 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:46.176469 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:46.176897 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:46.176992 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:46.180096 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:46.180110 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:46.180127 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:46.180160 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:46.182097 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:46.182704 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:46.182709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:46.182711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:46.182803 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:46.183103 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:46.183105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:46.183106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:46.183411 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:46.183571 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:46.184230 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 13:39:46.184420 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":91,"threads_started":1}
W20260926 13:39:46.185587 1377 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50292
Sep 26 13:39:46 dist-test-slave-1wns krb5kdc[1227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:39:46 dist-test-slave-1wns krb5kdc[1227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.1@KRBTEST.COM
I20260926 13:39:46.216192 1738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:46.206174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50306 (local address 127.1.49.1:45343)
0926 13:39:46.206319 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:46.206323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:46.206398 (+ 75us) server_negotiation.cc:407] Connection header received
0926 13:39:46.206525 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:46.206527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:46.206573 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:46.206663 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:46.207539 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:46.208166 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:46.209175 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:46.209405 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:46.209502 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:46.212534 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:46.212537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:46.212539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:46.212568 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:46.214182 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:46.214718 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:46.214720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:46.214722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:46.214803 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:46.215115 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:46.215117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:46.215118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:46.215298 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:46.215424 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:46.215922 (+ 498us) server_negotiation.cc:299] Negotiation successful
0926 13:39:46.216057 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:39:46.217478 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 1334
I20260926 13:39:46.223528 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 1468
I20260926 13:39:46.231345 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 1602
I20260926 13:39:46.237105 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 1240
2026-09-26T13:39:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2390 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:39:46 dist-test-slave-1wns krb5kdc[1745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:39:46 dist-test-slave-1wns krb5kdc[1745](info): set up 2 sockets
Sep 26 13:39:46 dist-test-slave-1wns krb5kdc[1745](info): commencing operation
krb5kdc: starting...
W20260926 13:39:47.320216 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:39:47 dist-test-slave-1wns krb5kdc[1745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:39:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:39:47.484536 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:43385
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:43385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:47.602455 1758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:47.602835 1758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:47.602922 1758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:47.607303 1758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:39:47.607445 1758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:47.607486 1758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:47.607524 1758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:39:47.607676 1758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:39:47.614048 1758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:43385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:43385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:47.615547 1758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:47.616945 1758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:47.628803 1766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:47.628803 1763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:47.628953 1764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:47.629365 1758 server_base.cc:1034] running on GCE node
I20260926 13:39:47.630170 1758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:47.631645 1758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:47.632915 1758 hybrid_clock.cc:648] HybridClock initialized: now 1790429987632876 us; error 58 us; skew 500 ppm
Sep 26 13:39:47 dist-test-slave-1wns krb5kdc[1745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429987, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:47.646013 1758 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:39:47.647260 1758 webserver.cc:460] Webserver started at http://127.1.49.62:33659/ using document root <none> and password file <none>
I20260926 13:39:47.648126 1758 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:47.648222 1758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:47.648497 1758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:47.651110 1758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "ce4f189f20984fbd8a710af7ee419118"
format_stamp: "Formatted at 2026-09-26 13:39:47 on dist-test-slave-1wns"
server_key: "424e856a62aba5d74997e8f37a7d2fde"
server_key_iv: "f206d1277d94c101816076a64a650fd3"
server_key_version: "encryptionkey@0"
I20260926 13:39:47.651957 1758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "ce4f189f20984fbd8a710af7ee419118"
format_stamp: "Formatted at 2026-09-26 13:39:47 on dist-test-slave-1wns"
server_key: "424e856a62aba5d74997e8f37a7d2fde"
server_key_iv: "f206d1277d94c101816076a64a650fd3"
server_key_version: "encryptionkey@0"
I20260926 13:39:47.656563 1758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:39:47.659754 1773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:47.661583 1758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:39:47.661761 1758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "ce4f189f20984fbd8a710af7ee419118"
format_stamp: "Formatted at 2026-09-26 13:39:47 on dist-test-slave-1wns"
server_key: "424e856a62aba5d74997e8f37a7d2fde"
server_key_iv: "f206d1277d94c101816076a64a650fd3"
server_key_version: "encryptionkey@0"
I20260926 13:39:47.661914 1758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:47.676350 1758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:47.680866 1758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:47.681118 1758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:47.689481 1758 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:43385
I20260926 13:39:47.689483 1825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:43385 every 8 connection(s)
I20260926 13:39:47.690671 1758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:39:47.692394 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 1758
I20260926 13:39:47.692651 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:39:47.692950 1220 external_mini_cluster.cc:1428] Setting key 6864af4048818ffd63bdc2d9505705f4
I20260926 13:39:47.694607 1826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:47.702011 1826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118: Bootstrap starting.
Sep 26 13:39:47 dist-test-slave-1wns krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:47.705193 1826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:47.706027 1826 log.cc:824] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118: Log is configured to *not* fsync() on all Append() calls
I20260926 13:39:47.709087 1826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118: No bootstrap required, opened a new log
I20260926 13:39:47.711127 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:47.695057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46774 (local address 127.1.49.62:43385)
0926 13:39:47.695561 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:47.695579 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:47.695617 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:39:47.696496 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:47.696515 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:47.696897 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:47.697237 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:47.698229 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:47.699295 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:47.700600 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:47.701165 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:47.701314 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:47.704604 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:47.704611 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:47.704629 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:47.704671 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:47.707323 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:47.708148 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:47.708154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:47.708161 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:47.708302 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:47.708718 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:47.708721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:47.708728 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:47.709247 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:47.709446 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:47.709865 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 13:39:47.710246 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":187,"threads_started":1}
I20260926 13:39:47.713512 1826 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce4f189f20984fbd8a710af7ee419118" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 43385 } }
I20260926 13:39:47.713922 1826 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:47.713992 1826 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4f189f20984fbd8a710af7ee419118, State: Initialized, Role: FOLLOWER
I20260926 13:39:47.714465 1826 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce4f189f20984fbd8a710af7ee419118" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 43385 } }
I20260926 13:39:47.714659 1826 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:39:47.714757 1826 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:39:47.714864 1826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:47.716059 1826 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce4f189f20984fbd8a710af7ee419118" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 43385 } }
I20260926 13:39:47.716466 1826 leader_election.cc:302] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce4f189f20984fbd8a710af7ee419118; no voters:
I20260926 13:39:47.716872 1826 leader_election.cc:288] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:39:47.717005 1831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:39:47.717336 1831 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [term 1 LEADER]: Becoming Leader. State: Replica: ce4f189f20984fbd8a710af7ee419118, State: Running, Role: LEADER
I20260926 13:39:47.717756 1831 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce4f189f20984fbd8a710af7ee419118" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 43385 } }
I20260926 13:39:47.718312 1826 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:39:47.719949 1833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce4f189f20984fbd8a710af7ee419118. Latest consensus state: current_term: 1 leader_uuid: "ce4f189f20984fbd8a710af7ee419118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce4f189f20984fbd8a710af7ee419118" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 43385 } } }
I20260926 13:39:47.720098 1833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [sys.catalog]: This master's current role is: LEADER
I20260926 13:39:47.720539 1840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:39:47.720844 1832 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce4f189f20984fbd8a710af7ee419118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce4f189f20984fbd8a710af7ee419118" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 43385 } } }
I20260926 13:39:47.721390 1832 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118 [sys.catalog]: This master's current role is: LEADER
I20260926 13:39:47.724598 1840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:39:47.732102 1840 catalog_manager.cc:1347] Generated new cluster ID: 72561850af214343ab4a1a81ecddb1d5
I20260926 13:39:47.732276 1840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:39:47.771147 1840 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:39:47.772770 1840 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:39:47.778057 1840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce4f189f20984fbd8a710af7ee419118: Generated new TSK 0
I20260926 13:39:47.779037 1840 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:39:47.824272 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:43385
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:47.945876 1852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:47.946205 1852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:47.946254 1852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:47.951105 1852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:47.951205 1852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:47.951279 1852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:39:47.957358 1852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:43385
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:47.958691 1852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:47.959923 1852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:47.971184 1860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:47.971246 1858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:47.971284 1852 server_base.cc:1034] running on GCE node
W20260926 13:39:47.971194 1857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:47.972029 1852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:47.972790 1852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:47.974007 1852 hybrid_clock.cc:648] HybridClock initialized: now 1790429987973977 us; error 43 us; skew 500 ppm
Sep 26 13:39:47 dist-test-slave-1wns krb5kdc[1745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429987, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:47.977522 1852 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:39:47.978571 1852 webserver.cc:460] Webserver started at http://127.1.49.1:33751/ using document root <none> and password file <none>
I20260926 13:39:47.979329 1852 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:47.979436 1852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:47.979686 1852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:47.982023 1852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "082490f4e1cb499e94be7f34febc3afd"
format_stamp: "Formatted at 2026-09-26 13:39:47 on dist-test-slave-1wns"
server_key: "c79a13a9c3e611a6dc4706b41fd08eea"
server_key_iv: "6bb4b2f8417b8a19d81a2819bb9c2cfe"
server_key_version: "encryptionkey@0"
I20260926 13:39:47.982645 1852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "082490f4e1cb499e94be7f34febc3afd"
format_stamp: "Formatted at 2026-09-26 13:39:47 on dist-test-slave-1wns"
server_key: "c79a13a9c3e611a6dc4706b41fd08eea"
server_key_iv: "6bb4b2f8417b8a19d81a2819bb9c2cfe"
server_key_version: "encryptionkey@0"
I20260926 13:39:47.986745 1852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 13:39:47.989784 1867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:47.991257 1852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:39:47.991441 1852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "082490f4e1cb499e94be7f34febc3afd"
format_stamp: "Formatted at 2026-09-26 13:39:47 on dist-test-slave-1wns"
server_key: "c79a13a9c3e611a6dc4706b41fd08eea"
server_key_iv: "6bb4b2f8417b8a19d81a2819bb9c2cfe"
server_key_version: "encryptionkey@0"
I20260926 13:39:47.991582 1852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:48.021929 1852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:48.025709 1852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:48.025949 1852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:48.026616 1852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:39:48.027572 1852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:39:48.027648 1852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:48.027722 1852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:39:48.027774 1852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:48.039290 1852 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:41165
I20260926 13:39:48.039350 1980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:41165 every 8 connection(s)
I20260926 13:39:48.040627 1852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:39:48.042878 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 1852
I20260926 13:39:48.042996 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:39:48.043299 1220 external_mini_cluster.cc:1428] Setting key edb03983e9cc3b8cf66d2c9e35faa4c0
Sep 26 13:39:48 dist-test-slave-1wns krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429987, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
I20260926 13:39:48.058840 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.043033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:55055 (local address 127.1.49.62:43385)
0926 13:39:48.043275 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:48.043279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:48.044515 (+ 1236us) server_negotiation.cc:407] Connection header received
0926 13:39:48.046073 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:48.046076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:48.046137 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:48.046218 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:48.047560 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.048258 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.049553 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.049817 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.049937 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.053392 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:48.053395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:48.053398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:48.053428 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:48.055702 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:48.056575 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:48.056580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:48.056581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:48.056795 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:48.057402 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:48.057411 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:48.057413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:48.057708 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:48.057831 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:48.058498 (+ 667us) server_negotiation.cc:299] Negotiation successful
0926 13:39:48.058662 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260926 13:39:48.060280 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.043687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:43385 (local address 127.1.49.1:55055)
0926 13:39:48.044359 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:48.044410 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:48.045827 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:48.046415 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:48.046423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:48.046825 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:48.047385 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.047401 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.048411 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.048414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.049400 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.049411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.049998 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.050002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.050233 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.051036 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:48.051055 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:48.053129 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:48.055928 (+ 2799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:48.055945 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:48.055963 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:48.056365 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:48.056919 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:48.056925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:48.056927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:48.057168 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:48.057832 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:48.057843 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:48.058211 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 13:39:48.058457 (+ 246us) client_negotiation.cc:240] Negotiation successful
0926 13:39:48.058893 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":134,"threads_started":1}
I20260926 13:39:48.062287 1981 heartbeater.cc:347] Connected to a master server at 127.1.49.62:43385
I20260926 13:39:48.062610 1981 heartbeater.cc:464] Registering TS with master...
I20260926 13:39:48.063248 1981 heartbeater.cc:511] Master 127.1.49.62:43385 requested a full tablet report, sending...
I20260926 13:39:48.065394 1790 ts_manager.cc:194] Registered new tserver with Master: 082490f4e1cb499e94be7f34febc3afd (127.1.49.1:41165)
I20260926 13:39:48.067248 1790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:55055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:39:48.081103 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:43385
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:48.203229 1986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:48.203567 1986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:48.203624 1986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:48.208068 1986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:48.208146 1986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:48.208247 1986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:39:48.214061 1986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:43385
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:48.215292 1986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:48.216537 1986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:48.227674 1994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:48.227674 1991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:48.227722 1986 server_base.cc:1034] running on GCE node
W20260926 13:39:48.228039 1992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:48.228641 1986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:48.229310 1986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:48.230484 1986 hybrid_clock.cc:648] HybridClock initialized: now 1790429988230410 us; error 85 us; skew 500 ppm
Sep 26 13:39:48 dist-test-slave-1wns krb5kdc[1745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429988, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:48.233845 1986 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:39:48.234928 1986 webserver.cc:460] Webserver started at http://127.1.49.2:38773/ using document root <none> and password file <none>
I20260926 13:39:48.235591 1986 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:48.235692 1986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:48.235996 1986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:48.238261 1986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "8f7a496be5db4fecbc7644bc18da1dd9"
format_stamp: "Formatted at 2026-09-26 13:39:48 on dist-test-slave-1wns"
server_key: "8e4a2cbc6ed0758e8168b333baf1137b"
server_key_iv: "6213031730d2ecd6e56f62609a041961"
server_key_version: "encryptionkey@0"
I20260926 13:39:48.238912 1986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "8f7a496be5db4fecbc7644bc18da1dd9"
format_stamp: "Formatted at 2026-09-26 13:39:48 on dist-test-slave-1wns"
server_key: "8e4a2cbc6ed0758e8168b333baf1137b"
server_key_iv: "6213031730d2ecd6e56f62609a041961"
server_key_version: "encryptionkey@0"
I20260926 13:39:48.243251 1986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 13:39:48.246335 2001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:48.248208 1986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:39:48.248389 1986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "8f7a496be5db4fecbc7644bc18da1dd9"
format_stamp: "Formatted at 2026-09-26 13:39:48 on dist-test-slave-1wns"
server_key: "8e4a2cbc6ed0758e8168b333baf1137b"
server_key_iv: "6213031730d2ecd6e56f62609a041961"
server_key_version: "encryptionkey@0"
I20260926 13:39:48.248538 1986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:48.271965 1986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:48.275933 1986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:48.276182 1986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:48.276942 1986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:39:48.277930 1986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:39:48.278002 1986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:48.278079 1986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:39:48.278120 1986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:48.288532 1986 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:33645
I20260926 13:39:48.288547 2114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:33645 every 8 connection(s)
I20260926 13:39:48.289565 1986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:39:48.299172 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 1986
I20260926 13:39:48.299329 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:39:48.299626 1220 external_mini_cluster.cc:1428] Setting key a460069644fa5fa4ab42991990db3951
Sep 26 13:39:48 dist-test-slave-1wns krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429988, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:48.308539 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.291923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:38197 (local address 127.1.49.62:43385)
0926 13:39:48.292077 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:48.292082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:48.293177 (+ 1095us) server_negotiation.cc:407] Connection header received
0926 13:39:48.294787 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:48.294791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:48.294852 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:48.294942 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:48.296735 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.297304 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.298776 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.299038 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.299148 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.303355 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:48.303362 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:48.303368 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:48.303409 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:48.305305 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:48.306238 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:48.306241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:48.306243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:48.306357 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:48.306917 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:48.306920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:48.306921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:48.307199 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:48.307315 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:48.308269 (+ 954us) server_negotiation.cc:299] Negotiation successful
0926 13:39:48.308407 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 13:39:48.310310 2117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.292342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:43385 (local address 127.1.49.2:38197)
0926 13:39:48.292984 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:48.293037 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:48.294482 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:48.295247 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:48.295266 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:48.295741 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:48.296476 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.296496 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.297507 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.297510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.298504 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.298532 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.299354 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.299361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.299627 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.300643 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:48.300667 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:48.303109 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:48.305471 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:48.305483 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:48.305508 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:48.306010 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:48.306473 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:48.306479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:48.306484 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:48.306755 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:48.307344 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:48.307355 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:48.307853 (+ 498us) client_negotiation.cc:769] Sending connection context
0926 13:39:48.308125 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 13:39:48.308579 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":159,"threads_started":1}
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
I20260926 13:39:48.312055 2115 heartbeater.cc:347] Connected to a master server at 127.1.49.62:43385
I20260926 13:39:48.312332 2115 heartbeater.cc:464] Registering TS with master...
I20260926 13:39:48.313133 2115 heartbeater.cc:511] Master 127.1.49.62:43385 requested a full tablet report, sending...
I20260926 13:39:48.314569 1790 ts_manager.cc:194] Registered new tserver with Master: 8f7a496be5db4fecbc7644bc18da1dd9 (127.1.49.2:33645)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
I20260926 13:39:48.315510 1790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:38197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:39:48.336719 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:43385
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:48.468400 2120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:48.468730 2120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:48.468779 2120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:48.473737 2120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:48.473827 2120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:48.473896 2120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:39:48.480186 2120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:43385
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:48.481506 2120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:48.482702 2120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:48.494614 2126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:48.494832 2120 server_base.cc:1034] running on GCE node
W20260926 13:39:48.494596 2125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:48.494596 2128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:48.495595 2120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:48.496479 2120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:48.497727 2120 hybrid_clock.cc:648] HybridClock initialized: now 1790429988497671 us; error 70 us; skew 500 ppm
Sep 26 13:39:48 dist-test-slave-1wns krb5kdc[1745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429988, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:48.502267 2120 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:39:48.503409 2120 webserver.cc:460] Webserver started at http://127.1.49.3:39933/ using document root <none> and password file <none>
I20260926 13:39:48.504276 2120 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:48.504390 2120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:48.504657 2120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:48.507009 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "b284513f18ec425587d933950683c4ac"
format_stamp: "Formatted at 2026-09-26 13:39:48 on dist-test-slave-1wns"
server_key: "2d968e41e03f2bb6e9fb35ccb09ea996"
server_key_iv: "0088b4ee8a802c261eaa0149857abee1"
server_key_version: "encryptionkey@0"
I20260926 13:39:48.507705 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "b284513f18ec425587d933950683c4ac"
format_stamp: "Formatted at 2026-09-26 13:39:48 on dist-test-slave-1wns"
server_key: "2d968e41e03f2bb6e9fb35ccb09ea996"
server_key_iv: "0088b4ee8a802c261eaa0149857abee1"
server_key_version: "encryptionkey@0"
I20260926 13:39:48.512439 2120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:39:48.516212 2135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:48.518217 2120 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:39:48.518402 2120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "b284513f18ec425587d933950683c4ac"
format_stamp: "Formatted at 2026-09-26 13:39:48 on dist-test-slave-1wns"
server_key: "2d968e41e03f2bb6e9fb35ccb09ea996"
server_key_iv: "0088b4ee8a802c261eaa0149857abee1"
server_key_version: "encryptionkey@0"
I20260926 13:39:48.518556 2120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:48.538132 2120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:48.542785 2120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:48.543134 2120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:48.543917 2120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:39:48.544873 2120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:39:48.544950 2120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:48.545023 2120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:39:48.545068 2120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:48.555186 2120 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:33507
I20260926 13:39:48.555255 2248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:33507 every 8 connection(s)
I20260926 13:39:48.556334 2120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:39:48.565405 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 2120
I20260926 13:39:48.565548 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:39:48.565887 1220 external_mini_cluster.cc:1428] Setting key 07bca46bca15019cc3d11fe69ab483bc
Sep 26 13:39:48 dist-test-slave-1wns krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429988, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:48.573036 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.558559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:59483 (local address 127.1.49.62:43385)
0926 13:39:48.558709 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:48.558713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:48.559715 (+ 1002us) server_negotiation.cc:407] Connection header received
0926 13:39:48.561406 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:48.561410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:48.561481 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:48.561560 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:48.562880 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.563422 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.564699 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.564923 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.565012 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.568625 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:48.568630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:48.568638 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:48.568686 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:48.570420 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:48.571128 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:48.571131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:48.571132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:48.571254 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:48.571824 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:48.571828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:48.571829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:48.572043 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:48.572129 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:48.572771 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 13:39:48.572901 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 13:39:48.574365 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.558941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:43385 (local address 127.1.49.3:59483)
0926 13:39:48.559523 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:48.559575 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:48.561072 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:48.561774 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:48.561783 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:48.562182 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:48.562725 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.562739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.563542 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.563544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.564540 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.564550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.565068 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.565071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.565386 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.566171 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:48.566195 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:48.568394 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:48.570570 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:48.570577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:48.570593 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:48.571006 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:48.571369 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:48.571372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:48.571374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:48.571640 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:48.572171 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:48.572178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:48.572499 (+ 321us) client_negotiation.cc:769] Sending connection context
0926 13:39:48.572684 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 13:39:48.573322 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":152,"threads_started":1}
I20260926 13:39:48.575897 2249 heartbeater.cc:347] Connected to a master server at 127.1.49.62:43385
I20260926 13:39:48.576400 2249 heartbeater.cc:464] Registering TS with master...
I20260926 13:39:48.577114 2249 heartbeater.cc:511] Master 127.1.49.62:43385 requested a full tablet report, sending...
I20260926 13:39:48.578891 1790 ts_manager.cc:194] Registered new tserver with Master: b284513f18ec425587d933950683c4ac (127.1.49.3:33507)
I20260926 13:39:48.579659 1790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:59483
I20260926 13:39:48.581415 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:39:48.595541 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.585558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46784 (local address 127.1.49.62:43385)
0926 13:39:48.585810 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:48.585814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:48.586004 (+ 190us) server_negotiation.cc:407] Connection header received
0926 13:39:48.586232 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:48.586235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:48.586286 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:48.586368 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:48.587316 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.588048 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.589189 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.589420 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.589525 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.591263 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:48.591266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:48.591268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:48.591301 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:48.592983 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:48.593525 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:48.593528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:48.593529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:48.593711 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:48.594103 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:48.594107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:48.594108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:48.594383 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:48.594524 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:48.595150 (+ 626us) server_negotiation.cc:299] Negotiation successful
0926 13:39:48.595381 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260926 13:39:48.602389 1790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:39:48.605897 1790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:39:48.623944 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.616528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:33645 (local address 127.0.0.1:39128)
0926 13:39:48.617745 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:48.617761 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:48.617897 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:48.618848 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:48.618851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:48.618866 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:48.618943 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.618950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.620341 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.620347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.621959 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.621965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.623192 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.623195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.623375 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.623425 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 13:39:48.623538 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 13:39:48.623734 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"mutex_wait_us":7,"spinlock_wait_cycles":1664,"thread_start_us":137,"threads_started":1}
I20260926 13:39:48.624631 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.616522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.1.49.1:41165)
0926 13:39:48.618152 (+ 1630us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:48.618160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:48.618182 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:39:48.618263 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:48.618268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:48.618475 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:48.619064 (+ 589us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:48.619242 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.620627 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.622116 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.622998 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.623115 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.623312 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:48.624393 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0926 13:39:48.624520 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1476,"thread_start_us":132,"threads_started":1}
I20260926 13:39:48.624753 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.616989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.1.49.2:33645)
0926 13:39:48.617381 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:48.617394 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:48.618050 (+ 656us) server_negotiation.cc:407] Connection header received
0926 13:39:48.618153 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:48.618160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:48.618476 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:48.618656 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:48.619081 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.620190 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.622115 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.622998 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.624003 (+ 1005us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.624408 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:48.624479 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 13:39:48.624597 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":166,"threads_started":1}
I20260926 13:39:48.624686 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.616389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:41165 (local address 127.0.0.1:48306)
0926 13:39:48.616911 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:48.616970 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:48.617141 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:48.618680 (+ 1539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:48.618687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:48.618715 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:48.618877 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.618885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.620766 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.620769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.621945 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.621960 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.623352 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.623355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.623585 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.623647 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 13:39:48.623731 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 13:39:48.623853 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":145,"thread_start_us":122,"threads_started":1}
I20260926 13:39:48.624831 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.617525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51322 (local address 127.1.49.3:33507)
0926 13:39:48.618247 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:48.618260 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:48.618285 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:39:48.618375 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:48.618380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:48.618591 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:48.618815 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:48.619276 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.620686 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.621962 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.623004 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.623137 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.623310 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:48.623781 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 13:39:48.623915 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":342,"threads_started":1}
I20260926 13:39:48.624361 2262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.617291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:33507 (local address 127.0.0.1:51322)
0926 13:39:48.617778 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:48.617796 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:48.617897 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:48.618969 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:48.618971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:48.618988 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:48.619129 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.619160 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.620814 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.620818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.621797 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.621809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.623244 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.623248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.623589 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.623641 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 13:39:48.623731 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 13:39:48.624114 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":333,"spinlock_wait_cycles":10240,"thread_start_us":103,"threads_started":1}
I20260926 13:39:48.628228 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 49cc26419f3a4702a2d4d6fa330a9f33 (DEFAULT_TABLE table=test-table [id=9595ed5a3ee44e168970c72dcfac8669]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:48.628209 1912 tablet_service.cc:1467] Processing CreateTablet for tablet 49cc26419f3a4702a2d4d6fa330a9f33 (DEFAULT_TABLE table=test-table [id=9595ed5a3ee44e168970c72dcfac8669]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:48.628211 2049 tablet_service.cc:1467] Processing CreateTablet for tablet 49cc26419f3a4702a2d4d6fa330a9f33 (DEFAULT_TABLE table=test-table [id=9595ed5a3ee44e168970c72dcfac8669]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:48.629400 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cc26419f3a4702a2d4d6fa330a9f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:48.629401 2049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cc26419f3a4702a2d4d6fa330a9f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:48.629401 1912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cc26419f3a4702a2d4d6fa330a9f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:48.637094 2266 tablet_bootstrap.cc:492] T 49cc26419f3a4702a2d4d6fa330a9f33 P b284513f18ec425587d933950683c4ac: Bootstrap starting.
I20260926 13:39:48.638139 2267 tablet_bootstrap.cc:492] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd: Bootstrap starting.
I20260926 13:39:48.640030 2266 tablet_bootstrap.cc:654] T 49cc26419f3a4702a2d4d6fa330a9f33 P b284513f18ec425587d933950683c4ac: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:48.640378 2268 tablet_bootstrap.cc:492] T 49cc26419f3a4702a2d4d6fa330a9f33 P 8f7a496be5db4fecbc7644bc18da1dd9: Bootstrap starting.
I20260926 13:39:48.640816 2267 tablet_bootstrap.cc:654] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:48.641733 2266 log.cc:824] T 49cc26419f3a4702a2d4d6fa330a9f33 P b284513f18ec425587d933950683c4ac: Log is configured to *not* fsync() on all Append() calls
I20260926 13:39:48.641733 2267 log.cc:824] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd: Log is configured to *not* fsync() on all Append() calls
I20260926 13:39:48.643359 2268 tablet_bootstrap.cc:654] T 49cc26419f3a4702a2d4d6fa330a9f33 P 8f7a496be5db4fecbc7644bc18da1dd9: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:48.644490 2268 log.cc:824] T 49cc26419f3a4702a2d4d6fa330a9f33 P 8f7a496be5db4fecbc7644bc18da1dd9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:39:48.644649 2267 tablet_bootstrap.cc:492] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd: No bootstrap required, opened a new log
I20260926 13:39:48.644837 2266 tablet_bootstrap.cc:492] T 49cc26419f3a4702a2d4d6fa330a9f33 P b284513f18ec425587d933950683c4ac: No bootstrap required, opened a new log
I20260926 13:39:48.644881 2267 ts_tablet_manager.cc:1397] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 13:39:48.645071 2266 ts_tablet_manager.cc:1397] T 49cc26419f3a4702a2d4d6fa330a9f33 P b284513f18ec425587d933950683c4ac: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260926 13:39:48.648619 2266 raft_consensus.cc:348] T 49cc26419f3a4702a2d4d6fa330a9f33 P b284513f18ec425587d933950683c4ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082490f4e1cb499e94be7f34febc3afd" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 41165 } } peers { permanent_uuid: "b284513f18ec425587d933950683c4ac" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 33507 } } peers { permanent_uuid: "8f7a496be5db4fecbc7644bc18da1dd9" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 33645 } }
I20260926 13:39:48.648983 2266 raft_consensus.cc:374] T 49cc26419f3a4702a2d4d6fa330a9f33 P b284513f18ec425587d933950683c4ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:48.649040 2266 raft_consensus.cc:729] T 49cc26419f3a4702a2d4d6fa330a9f33 P b284513f18ec425587d933950683c4ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b284513f18ec425587d933950683c4ac, State: Initialized, Role: FOLLOWER
I20260926 13:39:48.649159 2267 raft_consensus.cc:348] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082490f4e1cb499e94be7f34febc3afd" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 41165 } } peers { permanent_uuid: "b284513f18ec425587d933950683c4ac" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 33507 } } peers { permanent_uuid: "8f7a496be5db4fecbc7644bc18da1dd9" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 33645 } }
I20260926 13:39:48.649453 2267 raft_consensus.cc:374] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:48.649510 2267 raft_consensus.cc:729] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082490f4e1cb499e94be7f34febc3afd, State: Initialized, Role: FOLLOWER
I20260926 13:39:48.649611 2266 consensus_queue.cc:260] T 49cc26419f3a4702a2d4d6fa330a9f33 P b284513f18ec425587d933950683c4ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082490f4e1cb499e94be7f34febc3afd" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 41165 } } peers { permanent_uuid: "b284513f18ec425587d933950683c4ac" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 33507 } } peers { permanent_uuid: "8f7a496be5db4fecbc7644bc18da1dd9" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 33645 } }
I20260926 13:39:48.650046 2267 consensus_queue.cc:260] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082490f4e1cb499e94be7f34febc3afd" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 41165 } } peers { permanent_uuid: "b284513f18ec425587d933950683c4ac" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 33507 } } peers { permanent_uuid: "8f7a496be5db4fecbc7644bc18da1dd9" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 33645 } }
I20260926 13:39:48.650544 2249 heartbeater.cc:503] Master 127.1.49.62:43385 was elected leader, sending a full tablet report...
I20260926 13:39:48.650879 1981 heartbeater.cc:503] Master 127.1.49.62:43385 was elected leader, sending a full tablet report...
I20260926 13:39:48.651170 2267 ts_tablet_manager.cc:1428] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:39:48.651440 2266 ts_tablet_manager.cc:1428] T 49cc26419f3a4702a2d4d6fa330a9f33 P b284513f18ec425587d933950683c4ac: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 13:39:48.651872 2268 tablet_bootstrap.cc:492] T 49cc26419f3a4702a2d4d6fa330a9f33 P 8f7a496be5db4fecbc7644bc18da1dd9: No bootstrap required, opened a new log
I20260926 13:39:48.652127 2268 ts_tablet_manager.cc:1397] T 49cc26419f3a4702a2d4d6fa330a9f33 P 8f7a496be5db4fecbc7644bc18da1dd9: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260926 13:39:48.657064 2268 raft_consensus.cc:348] T 49cc26419f3a4702a2d4d6fa330a9f33 P 8f7a496be5db4fecbc7644bc18da1dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082490f4e1cb499e94be7f34febc3afd" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 41165 } } peers { permanent_uuid: "b284513f18ec425587d933950683c4ac" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 33507 } } peers { permanent_uuid: "8f7a496be5db4fecbc7644bc18da1dd9" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 33645 } }
I20260926 13:39:48.657389 2268 raft_consensus.cc:374] T 49cc26419f3a4702a2d4d6fa330a9f33 P 8f7a496be5db4fecbc7644bc18da1dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:48.657472 2268 raft_consensus.cc:729] T 49cc26419f3a4702a2d4d6fa330a9f33 P 8f7a496be5db4fecbc7644bc18da1dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f7a496be5db4fecbc7644bc18da1dd9, State: Initialized, Role: FOLLOWER
I20260926 13:39:48.658067 2268 consensus_queue.cc:260] T 49cc26419f3a4702a2d4d6fa330a9f33 P 8f7a496be5db4fecbc7644bc18da1dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082490f4e1cb499e94be7f34febc3afd" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 41165 } } peers { permanent_uuid: "b284513f18ec425587d933950683c4ac" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 33507 } } peers { permanent_uuid: "8f7a496be5db4fecbc7644bc18da1dd9" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 33645 } }
I20260926 13:39:48.658859 2115 heartbeater.cc:503] Master 127.1.49.62:43385 was elected leader, sending a full tablet report...
I20260926 13:39:48.660455 2268 ts_tablet_manager.cc:1428] T 49cc26419f3a4702a2d4d6fa330a9f33 P 8f7a496be5db4fecbc7644bc18da1dd9: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260926 13:39:48.790983 2116 tablet.cc:2369] T 49cc26419f3a4702a2d4d6fa330a9f33 P 8f7a496be5db4fecbc7644bc18da1dd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:39:48.792392 1982 tablet.cc:2369] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:39:48.807700 2250 tablet.cc:2369] T 49cc26419f3a4702a2d4d6fa330a9f33 P b284513f18ec425587d933950683c4ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:39:48.827975 2273 raft_consensus.cc:482] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:39:48.828228 2273 raft_consensus.cc:504] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082490f4e1cb499e94be7f34febc3afd" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 41165 } } peers { permanent_uuid: "b284513f18ec425587d933950683c4ac" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 33507 } } peers { permanent_uuid: "8f7a496be5db4fecbc7644bc18da1dd9" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 33645 } }
I20260926 13:39:48.829608 2273 leader_election.cc:288] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b284513f18ec425587d933950683c4ac (127.1.49.3:33507), 8f7a496be5db4fecbc7644bc18da1dd9 (127.1.49.2:33645)
I20260926 13:39:48.833853 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.830104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:59515 (local address 127.1.49.3:33507)
0926 13:39:48.830243 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:48.830248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:48.830437 (+ 189us) server_negotiation.cc:407] Connection header received
0926 13:39:48.830630 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:48.830633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:48.830682 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:48.830771 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:48.831048 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.831575 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.832840 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.833263 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.833385 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.833457 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:48.833645 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 13:39:48.833722 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:39:48.833877 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.829925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:33507 (local address 127.1.49.1:59515)
0926 13:39:48.830340 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:48.830353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:48.830513 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:48.830793 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:48.830796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:48.830812 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:48.830938 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.830943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.831695 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.831699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.832706 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.832715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.833391 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.833395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.833492 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.833516 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 13:39:48.833630 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 13:39:48.833708 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":45,"thread_start_us":113,"threads_started":1}
I20260926 13:39:48.834066 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.829930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:33645 (local address 127.1.49.1:36307)
0926 13:39:48.830459 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:48.830471 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:48.830780 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:48.831187 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:48.831190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:48.831207 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:48.831358 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.831363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.832129 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.832132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.832975 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:48.832984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.833628 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.833632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:48.833745 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.833762 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 13:39:48.833849 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 13:39:48.833919 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":70,"thread_start_us":127,"threads_started":1}
I20260926 13:39:48.834048 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.830032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:36307 (local address 127.1.49.2:33645)
0926 13:39:48.830190 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:48.830198 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:48.830562 (+ 364us) server_negotiation.cc:407] Connection header received
0926 13:39:48.830934 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:48.830938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:48.831058 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:48.831179 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:48.831471 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.832006 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.833104 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.833501 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.833623 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.833686 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:48.833866 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 13:39:48.833937 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 13:39:48.834779 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cc26419f3a4702a2d4d6fa330a9f33" candidate_uuid: "082490f4e1cb499e94be7f34febc3afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f7a496be5db4fecbc7644bc18da1dd9" is_pre_election: true
I20260926 13:39:48.834771 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cc26419f3a4702a2d4d6fa330a9f33" candidate_uuid: "082490f4e1cb499e94be7f34febc3afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b284513f18ec425587d933950683c4ac" is_pre_election: true
I20260926 13:39:48.835103 2203 raft_consensus.cc:2454] T 49cc26419f3a4702a2d4d6fa330a9f33 P b284513f18ec425587d933950683c4ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 082490f4e1cb499e94be7f34febc3afd in term 0.
I20260926 13:39:48.835104 2068 raft_consensus.cc:2454] T 49cc26419f3a4702a2d4d6fa330a9f33 P 8f7a496be5db4fecbc7644bc18da1dd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 082490f4e1cb499e94be7f34febc3afd in term 0.
I20260926 13:39:48.835705 1869 leader_election.cc:302] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 082490f4e1cb499e94be7f34febc3afd, 8f7a496be5db4fecbc7644bc18da1dd9, b284513f18ec425587d933950683c4ac; no voters:
I20260926 13:39:48.836295 2273 raft_consensus.cc:2781] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:39:48.836391 2273 raft_consensus.cc:482] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:39:48.836423 2273 raft_consensus.cc:3037] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:48.837580 2273 raft_consensus.cc:504] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082490f4e1cb499e94be7f34febc3afd" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 41165 } } peers { permanent_uuid: "b284513f18ec425587d933950683c4ac" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 33507 } } peers { permanent_uuid: "8f7a496be5db4fecbc7644bc18da1dd9" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 33645 } }
I20260926 13:39:48.838022 2273 leader_election.cc:288] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [CANDIDATE]: Term 1 election: Requested vote from peers b284513f18ec425587d933950683c4ac (127.1.49.3:33507), 8f7a496be5db4fecbc7644bc18da1dd9 (127.1.49.2:33645)
I20260926 13:39:48.838570 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cc26419f3a4702a2d4d6fa330a9f33" candidate_uuid: "082490f4e1cb499e94be7f34febc3afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f7a496be5db4fecbc7644bc18da1dd9"
I20260926 13:39:48.838570 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cc26419f3a4702a2d4d6fa330a9f33" candidate_uuid: "082490f4e1cb499e94be7f34febc3afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b284513f18ec425587d933950683c4ac"
I20260926 13:39:48.838724 2203 raft_consensus.cc:3037] T 49cc26419f3a4702a2d4d6fa330a9f33 P b284513f18ec425587d933950683c4ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:48.838816 2068 raft_consensus.cc:3037] T 49cc26419f3a4702a2d4d6fa330a9f33 P 8f7a496be5db4fecbc7644bc18da1dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:48.840219 2068 raft_consensus.cc:2454] T 49cc26419f3a4702a2d4d6fa330a9f33 P 8f7a496be5db4fecbc7644bc18da1dd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 082490f4e1cb499e94be7f34febc3afd in term 1.
I20260926 13:39:48.840219 2203 raft_consensus.cc:2454] T 49cc26419f3a4702a2d4d6fa330a9f33 P b284513f18ec425587d933950683c4ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 082490f4e1cb499e94be7f34febc3afd in term 1.
I20260926 13:39:48.840700 1869 leader_election.cc:302] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082490f4e1cb499e94be7f34febc3afd, b284513f18ec425587d933950683c4ac; no voters:
I20260926 13:39:48.841022 2273 raft_consensus.cc:2781] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:39:48.841388 2273 raft_consensus.cc:686] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [term 1 LEADER]: Becoming Leader. State: Replica: 082490f4e1cb499e94be7f34febc3afd, State: Running, Role: LEADER
I20260926 13:39:48.841848 2273 consensus_queue.cc:237] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082490f4e1cb499e94be7f34febc3afd" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 41165 } } peers { permanent_uuid: "b284513f18ec425587d933950683c4ac" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 33507 } } peers { permanent_uuid: "8f7a496be5db4fecbc7644bc18da1dd9" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 33645 } }
I20260926 13:39:48.845724 1790 catalog_manager.cc:5495] T 49cc26419f3a4702a2d4d6fa330a9f33 P 082490f4e1cb499e94be7f34febc3afd reported cstate change: term changed from 0 to 1, leader changed from <none> to 082490f4e1cb499e94be7f34febc3afd (127.1.49.1). New cstate: current_term: 1 leader_uuid: "082490f4e1cb499e94be7f34febc3afd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082490f4e1cb499e94be7f34febc3afd" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 41165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b284513f18ec425587d933950683c4ac" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f7a496be5db4fecbc7644bc18da1dd9" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 33645 } health_report { overall_health: UNKNOWN } } }
Sep 26 13:39:48 dist-test-slave-1wns krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.1@KRBTEST.COM
I20260926 13:39:48.869231 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.858345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.1.49.1:41165)
0926 13:39:48.858559 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:48.858564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:48.858723 (+ 159us) server_negotiation.cc:407] Connection header received
0926 13:39:48.858895 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:48.858898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:48.858955 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:48.859047 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:48.859987 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.860511 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.861510 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.861746 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.861862 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.865056 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:48.865072 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:48.865089 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:48.865165 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:48.867226 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:48.867901 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:48.867907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:48.867908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:48.868003 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:48.868316 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:48.868318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:48.868319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:48.868603 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:48.868844 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:48.868975 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 13:39:48.869090 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
Sep 26 13:39:48 dist-test-slave-1wns krb5kdc[1745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429988, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:39:48 dist-test-slave-1wns krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429988, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.49.1@KRBTEST.COM
I20260926 13:39:48.901260 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.890884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.1.49.1:41165)
0926 13:39:48.891046 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:48.891050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:48.891168 (+ 118us) server_negotiation.cc:407] Connection header received
0926 13:39:48.891231 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:48.891233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:48.891283 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:48.891367 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:48.892260 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.892880 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.893806 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.894051 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.894151 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.897213 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:48.897217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:48.897219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:48.897254 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:48.899028 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:48.899553 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:48.899557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:48.899558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:48.899657 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:48.900136 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:48.900139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:48.900140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:48.900349 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:48.900475 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:48.900912 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 13:39:48.901037 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 13:39:48 dist-test-slave-1wns krb5kdc[1745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:39:48 dist-test-slave-1wns krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.1@KRBTEST.COM
I20260926 13:39:48.931129 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:48.921192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.1.49.1:41165)
0926 13:39:48.921378 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:48.921385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:48.921487 (+ 102us) server_negotiation.cc:407] Connection header received
0926 13:39:48.921614 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:48.921616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:48.921671 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:48.921756 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:48.922533 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.923066 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.924194 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:48.924424 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:48.924532 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:48.927386 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:48.927389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:48.927391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:48.927422 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:48.929153 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:48.929683 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:48.929686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:48.929688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:48.929772 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:48.930241 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:48.930244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:48.930245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:48.930435 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:48.930580 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:48.930792 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 13:39:48.930905 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 13:39:48.932627 1895 tablet_service.cc:2908] Rejecting scan request for tablet 49cc26419f3a4702a2d4d6fa330a9f33: Uninitialized: clean time has not yet been initialized
I20260926 13:39:48.935046 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 1852
I20260926 13:39:48.941205 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 1986
I20260926 13:39:48.946830 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 2120
I20260926 13:39:48.952780 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 1758
2026-09-26T13:39:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2717 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:39:49 dist-test-slave-1wns krb5kdc[2291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:39:49 dist-test-slave-1wns krb5kdc[2291](info): set up 2 sockets
Sep 26 13:39:49 dist-test-slave-1wns krb5kdc[2291](info): commencing operation
krb5kdc: starting...
W20260926 13:39:50.032198 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:39:50 dist-test-slave-1wns krb5kdc[2291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:39:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:39:50.172919 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:37387
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:37387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:50.290540 2304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:50.290853 2304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:50.290894 2304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:50.295260 2304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:39:50.295336 2304 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 13:39:50.295353 2304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:50.295372 2304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:50.295418 2304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:39:50.295455 2304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:39:50.301322 2304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:37387
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:37387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:50.302536 2304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:50.303676 2304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:50.313776 2309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:50.313786 2312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:50.313998 2304 server_base.cc:1034] running on GCE node
W20260926 13:39:50.314301 2310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:50.315078 2304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:50.316571 2304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:50.317801 2304 hybrid_clock.cc:648] HybridClock initialized: now 1790429990317767 us; error 54 us; skew 500 ppm
Sep 26 13:39:50 dist-test-slave-1wns krb5kdc[2291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429990, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:50.321655 2304 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:39:50.322762 2304 webserver.cc:460] Webserver started at http://127.1.49.62:37823/ using document root <none> and password file <none>
I20260926 13:39:50.323480 2304 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:50.323560 2304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:50.323930 2304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:50.326269 2304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "ac6647ea3eaa4561b1d0b8ce208cf3ea"
format_stamp: "Formatted at 2026-09-26 13:39:50 on dist-test-slave-1wns"
server_key: "c63d64f0c176da0e494c2bd92f9c3c61"
server_key_iv: "d29c28a861ed90791a0a73563efcbcdc"
server_key_version: "encryptionkey@0"
I20260926 13:39:50.326896 2304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "ac6647ea3eaa4561b1d0b8ce208cf3ea"
format_stamp: "Formatted at 2026-09-26 13:39:50 on dist-test-slave-1wns"
server_key: "c63d64f0c176da0e494c2bd92f9c3c61"
server_key_iv: "d29c28a861ed90791a0a73563efcbcdc"
server_key_version: "encryptionkey@0"
I20260926 13:39:50.331166 2304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 13:39:50.333947 2319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:50.335500 2304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:39:50.335678 2304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "ac6647ea3eaa4561b1d0b8ce208cf3ea"
format_stamp: "Formatted at 2026-09-26 13:39:50 on dist-test-slave-1wns"
server_key: "c63d64f0c176da0e494c2bd92f9c3c61"
server_key_iv: "d29c28a861ed90791a0a73563efcbcdc"
server_key_version: "encryptionkey@0"
I20260926 13:39:50.335840 2304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:50.360211 2304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:50.364387 2304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:50.364625 2304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:50.373634 2304 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:37387
I20260926 13:39:50.373651 2381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:37387 every 8 connection(s)
I20260926 13:39:50.374719 2304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:39:50.378168 2382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:50.380775 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 2304
I20260926 13:39:50.380913 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:39:50.381192 1220 external_mini_cluster.cc:1428] Setting key ec174edaeb5cf024636601f305b6164b
I20260926 13:39:50.385703 2382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea: Bootstrap starting.
I20260926 13:39:50.388816 2382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:50.389849 2382 log.cc:824] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea: Log is configured to *not* fsync() on all Append() calls
Sep 26 13:39:50 dist-test-slave-1wns krb5kdc[2291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:50.392326 2382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea: No bootstrap required, opened a new log
I20260926 13:39:50.397670 2382 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6647ea3eaa4561b1d0b8ce208cf3ea" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 37387 } }
I20260926 13:39:50.398022 2382 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:50.398097 2382 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6647ea3eaa4561b1d0b8ce208cf3ea, State: Initialized, Role: FOLLOWER
I20260926 13:39:50.398660 2382 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6647ea3eaa4561b1d0b8ce208cf3ea" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 37387 } }
I20260926 13:39:50.398801 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:50.382897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.1.49.62:37387)
0926 13:39:50.383372 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:50.383405 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:50.383446 (+ 41us) server_negotiation.cc:407] Connection header received
0926 13:39:50.384420 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:50.384443 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:50.384898 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:50.385250 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:50.386272 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.387314 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.388717 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.389332 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.389483 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:50.392373 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:50.392383 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:50.392403 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:50.392439 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:50.394780 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:50.395416 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:50.395423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:50.395438 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:50.395613 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:50.396022 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:50.396026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:50.396029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:50.396660 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:50.396896 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:50.397247 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 13:39:50.397595 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":174,"threads_started":1}
I20260926 13:39:50.398855 2382 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:39:50.398917 2382 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:39:50.399015 2382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:50.400234 2382 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6647ea3eaa4561b1d0b8ce208cf3ea" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 37387 } }
I20260926 13:39:50.400642 2382 leader_election.cc:302] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac6647ea3eaa4561b1d0b8ce208cf3ea; no voters:
I20260926 13:39:50.401041 2382 leader_election.cc:288] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:39:50.401131 2387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:39:50.401372 2387 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [term 1 LEADER]: Becoming Leader. State: Replica: ac6647ea3eaa4561b1d0b8ce208cf3ea, State: Running, Role: LEADER
I20260926 13:39:50.401748 2387 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6647ea3eaa4561b1d0b8ce208cf3ea" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 37387 } }
I20260926 13:39:50.402530 2382 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:39:50.403086 2388 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac6647ea3eaa4561b1d0b8ce208cf3ea. Latest consensus state: current_term: 1 leader_uuid: "ac6647ea3eaa4561b1d0b8ce208cf3ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6647ea3eaa4561b1d0b8ce208cf3ea" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 37387 } } }
I20260926 13:39:50.403352 2388 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [sys.catalog]: This master's current role is: LEADER
I20260926 13:39:50.403617 2389 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac6647ea3eaa4561b1d0b8ce208cf3ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6647ea3eaa4561b1d0b8ce208cf3ea" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 37387 } } }
I20260926 13:39:50.403744 2389 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea [sys.catalog]: This master's current role is: LEADER
I20260926 13:39:50.403872 2393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:39:50.407217 2393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:39:50.413234 2393 catalog_manager.cc:1347] Generated new cluster ID: f3f08851ce35447a9622731faed80f89
I20260926 13:39:50.413327 2393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:39:50.429908 2393 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:39:50.431094 2393 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:39:50.438902 2393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea: Generated new TSK 0
I20260926 13:39:50.439785 2393 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:39:50.480815 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:37387
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:50.603010 2408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:50.603335 2408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:50.603443 2408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:50.608363 2408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:50.608448 2408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:50.608491 2408 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:39:50.608567 2408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:39:50.614321 2408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.49.62:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:50.615540 2408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:50.616731 2408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:50.627805 2416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:50.627871 2414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:50.627871 2413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:50.628500 2408 server_base.cc:1034] running on GCE node
I20260926 13:39:50.629163 2408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:50.629957 2408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:50.631196 2408 hybrid_clock.cc:648] HybridClock initialized: now 1790429990631160 us; error 66 us; skew 500 ppm
Sep 26 13:39:50 dist-test-slave-1wns krb5kdc[2291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429990, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:50.634385 2408 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:39:50.635418 2408 webserver.cc:460] Webserver started at http://127.1.49.1:37657/ using document root <none> and password file <none>
I20260926 13:39:50.636108 2408 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:50.636214 2408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:50.636469 2408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:50.638774 2408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "8425208696124113bfb0af38b7fc32f6"
format_stamp: "Formatted at 2026-09-26 13:39:50 on dist-test-slave-1wns"
server_key: "896ca39652764ea71aae1860209cfd67"
server_key_iv: "e15cf7d527c2a6950cc9fd1bc829bc24"
server_key_version: "encryptionkey@0"
I20260926 13:39:50.639384 2408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "8425208696124113bfb0af38b7fc32f6"
format_stamp: "Formatted at 2026-09-26 13:39:50 on dist-test-slave-1wns"
server_key: "896ca39652764ea71aae1860209cfd67"
server_key_iv: "e15cf7d527c2a6950cc9fd1bc829bc24"
server_key_version: "encryptionkey@0"
I20260926 13:39:50.643869 2408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 13:39:50.646701 2423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:50.648468 2408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:39:50.648645 2408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "8425208696124113bfb0af38b7fc32f6"
format_stamp: "Formatted at 2026-09-26 13:39:50 on dist-test-slave-1wns"
server_key: "896ca39652764ea71aae1860209cfd67"
server_key_iv: "e15cf7d527c2a6950cc9fd1bc829bc24"
server_key_version: "encryptionkey@0"
I20260926 13:39:50.648790 2408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:50.671401 2408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:50.675298 2408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:50.675535 2408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:50.677157 2408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:39:50.677240 2408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:50.677312 2408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:39:50.677363 2408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:39:50 dist-test-slave-1wns krb5kdc[2291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429990, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:50.689512 2408 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:38567
I20260926 13:39:50.689531 2538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:38567 every 8 connection(s)
I20260926 13:39:50.690547 2408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:39:50.692715 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:50.678253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:49281 (local address 127.1.49.62:37387)
0926 13:39:50.678418 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:50.678422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:50.679462 (+ 1040us) server_negotiation.cc:407] Connection header received
0926 13:39:50.680601 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:50.680605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:50.680671 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:50.680773 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:50.682295 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.682874 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.684406 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.684759 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.684914 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:50.688527 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:50.688531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:50.688533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:50.688563 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:50.690255 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:50.690949 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:50.690952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:50.690954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:50.691073 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:50.691482 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:50.691485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:50.691487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:50.691698 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:50.691811 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:50.692382 (+ 571us) server_negotiation.cc:299] Negotiation successful
0926 13:39:50.692525 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 13:39:50.693975 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:50.678758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:37387 (local address 127.1.49.1:49281)
0926 13:39:50.679259 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:50.679314 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:50.680346 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:50.680991 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:50.681001 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:50.681447 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:50.682042 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:50.682110 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.683031 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.683034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:50.684259 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:50.684272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.684975 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.684979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:50.685175 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:50.685931 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:50.685963 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:50.688280 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:50.690412 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:50.690419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:50.690437 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:50.690818 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:50.691194 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:50.691197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:50.691199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:50.691365 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:50.691834 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:50.691844 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:50.692120 (+ 276us) client_negotiation.cc:769] Sending connection context
0926 13:39:50.692325 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 13:39:50.692676 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":141,"threads_started":1}
I20260926 13:39:50.695935 2539 heartbeater.cc:347] Connected to a master server at 127.1.49.62:37387
I20260926 13:39:50.696151 2539 heartbeater.cc:464] Registering TS with master...
I20260926 13:39:50.696761 2539 heartbeater.cc:511] Master 127.1.49.62:37387 requested a full tablet report, sending...
I20260926 13:39:50.698693 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 2408
I20260926 13:39:50.698762 2332 ts_manager.cc:194] Registered new tserver with Master: 8425208696124113bfb0af38b7fc32f6 (127.1.49.1:38567)
I20260926 13:39:50.698843 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:39:50.699115 1220 external_mini_cluster.cc:1428] Setting key a34689bc785c648d3084324a0ab6d74d
I20260926 13:39:50.700310 2332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:49281
I20260926 13:39:50.708757 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:50.698430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:37947 (local address 127.1.49.62:37387)
0926 13:39:50.699945 (+ 1515us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:50.699949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:50.699963 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:39:50.699999 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:50.700002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:50.700050 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:50.700131 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:50.700839 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.701517 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.702634 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.703099 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.703233 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:50.704784 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:50.704787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:50.704789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:50.704814 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:50.706617 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:50.707258 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:50.707261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:50.707263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:50.707362 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:50.707738 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:50.707741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:50.707742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:50.708042 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:50.708144 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:50.708492 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 13:39:50.708629 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1431}
I20260926 13:39:50.708874 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:50.698378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:37387 (local address 127.1.49.1:37947)
0926 13:39:50.698726 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:50.698743 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:50.698851 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:50.700185 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:50.700189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:50.700437 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:50.700694 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:50.700708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.701679 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.701683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:50.702493 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:50.702505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.703237 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.703241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:50.703331 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:50.704016 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:50.704036 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:50.704607 (+ 571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:50.706794 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:50.706800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:50.706804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:50.707112 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:50.707473 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:50.707478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:50.707483 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:50.707612 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:50.708163 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:50.708166 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:50.708291 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 13:39:50.708423 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 13:39:50.708621 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":138,"threads_started":1}
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:39:50.733080 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:37387
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:50.854928 2548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:50.855291 2548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:50.855372 2548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:50.859683 2548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:50.859812 2548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:50.859923 2548 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:39:50.860038 2548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:39:50.865952 2548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.49.62:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:50.867192 2548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:50.868463 2548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:50.879360 2553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:50.879347 2556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:50.879347 2554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:50.880131 2548 server_base.cc:1034] running on GCE node
I20260926 13:39:50.880563 2548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:50.881230 2548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:50.882416 2548 hybrid_clock.cc:648] HybridClock initialized: now 1790429990882390 us; error 36 us; skew 500 ppm
Sep 26 13:39:50 dist-test-slave-1wns krb5kdc[2291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429990, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:50.885860 2548 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:39:50.886878 2548 webserver.cc:460] Webserver started at http://127.1.49.2:34241/ using document root <none> and password file <none>
I20260926 13:39:50.887718 2548 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:50.887863 2548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:50.888123 2548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:50.890544 2548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "d63efd3ab1ee461391751867ba5a7616"
format_stamp: "Formatted at 2026-09-26 13:39:50 on dist-test-slave-1wns"
server_key: "eaa8fb59ec29fbbf3d585e1ec7f2babf"
server_key_iv: "c1b102e9a8187510f678134eabcb7dce"
server_key_version: "encryptionkey@0"
I20260926 13:39:50.891173 2548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "d63efd3ab1ee461391751867ba5a7616"
format_stamp: "Formatted at 2026-09-26 13:39:50 on dist-test-slave-1wns"
server_key: "eaa8fb59ec29fbbf3d585e1ec7f2babf"
server_key_iv: "c1b102e9a8187510f678134eabcb7dce"
server_key_version: "encryptionkey@0"
I20260926 13:39:50.895452 2548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 13:39:50.898639 2563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:50.900194 2548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:39:50.900357 2548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "d63efd3ab1ee461391751867ba5a7616"
format_stamp: "Formatted at 2026-09-26 13:39:50 on dist-test-slave-1wns"
server_key: "eaa8fb59ec29fbbf3d585e1ec7f2babf"
server_key_iv: "c1b102e9a8187510f678134eabcb7dce"
server_key_version: "encryptionkey@0"
I20260926 13:39:50.900494 2548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:50.915226 2548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:50.919131 2548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:50.919368 2548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:50.921104 2548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:39:50.921202 2548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:50.921271 2548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:39:50.921315 2548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:39:50 dist-test-slave-1wns krb5kdc[2291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429990, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:50.933290 2548 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:42193
I20260926 13:39:50.933362 2678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:42193 every 8 connection(s)
I20260926 13:39:50.934500 2548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:39:50.937274 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:50.922214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:44239 (local address 127.1.49.62:37387)
0926 13:39:50.922369 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:50.922372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:50.923713 (+ 1341us) server_negotiation.cc:407] Connection header received
0926 13:39:50.925050 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:50.925054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:50.925119 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:50.925259 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:50.926969 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.927893 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.929296 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.929547 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.929670 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:50.933182 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:50.933185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:50.933187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:50.933217 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:50.934818 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:50.935578 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:50.935582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:50.935586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:50.935701 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:50.936102 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:50.936106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:50.936107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:50.936309 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:50.936392 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:50.937011 (+ 619us) server_negotiation.cc:299] Negotiation successful
0926 13:39:50.937143 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 13:39:50.938220 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:50.922882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:37387 (local address 127.1.49.2:44239)
0926 13:39:50.923483 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:50.923592 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:50.924761 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:50.925508 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:50.925521 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:50.926032 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:50.926790 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:50.926811 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.928033 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.928036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:50.929156 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:50.929167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.929716 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.929720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:50.929967 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:50.930725 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:50.930742 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:50.932876 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:50.935001 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:50.935007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:50.935024 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:50.935450 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:50.935849 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:50.935852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:50.935854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:50.936007 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:50.936410 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:50.936415 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:50.936745 (+ 330us) client_negotiation.cc:769] Sending connection context
0926 13:39:50.936911 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 13:39:50.937194 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":163,"threads_started":1}
I20260926 13:39:50.939895 2679 heartbeater.cc:347] Connected to a master server at 127.1.49.62:37387
I20260926 13:39:50.940106 2679 heartbeater.cc:464] Registering TS with master...
I20260926 13:39:50.940861 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 2548
I20260926 13:39:50.940863 2679 heartbeater.cc:511] Master 127.1.49.62:37387 requested a full tablet report, sending...
I20260926 13:39:50.940968 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:39:50.941382 1220 external_mini_cluster.cc:1428] Setting key c082d173c603d19517727434edd89095
I20260926 13:39:50.942297 2332 ts_manager.cc:194] Registered new tserver with Master: d63efd3ab1ee461391751867ba5a7616 (127.1.49.2:42193)
I20260926 13:39:50.943050 2332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:44239
I20260926 13:39:50.950999 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:50.942433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:43657 (local address 127.1.49.62:37387)
0926 13:39:50.942577 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:50.942582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:50.943590 (+ 1008us) server_negotiation.cc:407] Connection header received
0926 13:39:50.943724 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:50.943727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:50.943778 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:50.943886 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:50.944394 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.944918 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.945796 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.946052 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.946167 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:50.947457 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:50.947461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:50.947463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:50.947498 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:50.949191 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:50.949670 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:50.949674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:50.949676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:50.949763 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:50.950057 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:50.950061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:50.950062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:50.950306 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:50.950421 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:50.950703 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 13:39:50.950820 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:39:50.951017 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:50.942395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:37387 (local address 127.1.49.2:43657)
0926 13:39:50.943486 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:50.943504 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:50.943621 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:50.943903 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:50.943906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:50.944123 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:50.944276 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:50.944283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.945052 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.945055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:50.945668 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:50.945677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:50.946172 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:50.946175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:50.946288 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:50.946875 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:50.946887 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:50.947294 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:50.949327 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:50.949331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:50.949333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:50.949560 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:50.949860 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:50.949863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:50.949865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:50.949957 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:50.950407 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:50.950410 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:50.950529 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 13:39:50.950646 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 13:39:50.950812 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"mutex_wait_us":20,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:39:50.972258 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:37387
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:51.093936 2688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:51.094254 2688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:51.094302 2688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:51.098850 2688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:51.098925 2688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:51.098968 2688 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:39:51.099004 2688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:39:51.105044 2688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.49.62:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:51.106258 2688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:51.107419 2688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:51.119782 2694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:51.119735 2696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:51.119743 2693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:51.119990 2688 server_base.cc:1034] running on GCE node
I20260926 13:39:51.120604 2688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:51.121385 2688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:51.122592 2688 hybrid_clock.cc:648] HybridClock initialized: now 1790429991122573 us; error 36 us; skew 500 ppm
Sep 26 13:39:51 dist-test-slave-1wns krb5kdc[2291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429991, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:51.126276 2688 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:39:51.127472 2688 webserver.cc:460] Webserver started at http://127.1.49.3:43765/ using document root <none> and password file <none>
I20260926 13:39:51.128211 2688 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:51.128290 2688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:51.128492 2688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:51.130995 2688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "302f09efedd44c2482017c0dad16b131"
format_stamp: "Formatted at 2026-09-26 13:39:51 on dist-test-slave-1wns"
server_key: "e2451a8c7f64ee9a10fd8d11cbb06673"
server_key_iv: "31f55c246efdc673fb714899c366e771"
server_key_version: "encryptionkey@0"
I20260926 13:39:51.131626 2688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "302f09efedd44c2482017c0dad16b131"
format_stamp: "Formatted at 2026-09-26 13:39:51 on dist-test-slave-1wns"
server_key: "e2451a8c7f64ee9a10fd8d11cbb06673"
server_key_iv: "31f55c246efdc673fb714899c366e771"
server_key_version: "encryptionkey@0"
I20260926 13:39:51.136392 2688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:39:51.139484 2703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:51.141346 2688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:39:51.141484 2688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "302f09efedd44c2482017c0dad16b131"
format_stamp: "Formatted at 2026-09-26 13:39:51 on dist-test-slave-1wns"
server_key: "e2451a8c7f64ee9a10fd8d11cbb06673"
server_key_iv: "31f55c246efdc673fb714899c366e771"
server_key_version: "encryptionkey@0"
I20260926 13:39:51.141655 2688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:51.159334 2688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:51.163085 2688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:51.163347 2688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:51.165103 2688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:39:51.165169 2688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:51.165243 2688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:39:51.165282 2688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:39:51 dist-test-slave-1wns krb5kdc[2291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429991, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:51.177858 2688 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:40553
I20260926 13:39:51.177933 2818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:40553 every 8 connection(s)
I20260926 13:39:51.179292 2688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:39:51.180181 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 2688
I20260926 13:39:51.180335 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:39:51.180627 1220 external_mini_cluster.cc:1428] Setting key c86f30a6554ec4b03ad7a73be19a4c59
I20260926 13:39:51.181731 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.166276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:58079 (local address 127.1.49.62:37387)
0926 13:39:51.166441 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.166446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.167497 (+ 1051us) server_negotiation.cc:407] Connection header received
0926 13:39:51.168788 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.168795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.168879 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.169044 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:51.170900 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.171780 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.173434 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.173693 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.173820 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.176821 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:51.176826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:51.176829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:51.176865 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:51.179010 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:51.179664 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:51.179671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:51.179676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:51.179791 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:51.180290 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:51.180293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:51.180295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:51.180577 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:51.180706 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.181397 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.181539 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 13:39:51.182662 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.166735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:37387 (local address 127.1.49.3:58079)
0926 13:39:51.167290 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.167349 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.168452 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.169253 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.169265 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.169760 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:51.170696 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.170724 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.171947 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.171950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.173291 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.173308 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.173882 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.173886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.174088 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.174774 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:51.174791 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:51.176594 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:51.179170 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:51.179175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:51.179188 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:51.179544 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:51.179957 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:51.179961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:51.179963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:51.180160 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:51.180706 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:51.180716 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:51.181067 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.181346 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.181726 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":162,"threads_started":1}
I20260926 13:39:51.184235 2819 heartbeater.cc:347] Connected to a master server at 127.1.49.62:37387
I20260926 13:39:51.186599 2819 heartbeater.cc:464] Registering TS with master...
I20260926 13:39:51.187291 2819 heartbeater.cc:511] Master 127.1.49.62:37387 requested a full tablet report, sending...
I20260926 13:39:51.188556 2332 ts_manager.cc:194] Registered new tserver with Master: 302f09efedd44c2482017c0dad16b131 (127.1.49.3:40553)
I20260926 13:39:51.189203 2332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:58079
I20260926 13:39:51.196105 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.186996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:48513 (local address 127.1.49.62:37387)
0926 13:39:51.187144 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.187150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.187515 (+ 365us) server_negotiation.cc:407] Connection header received
0926 13:39:51.187642 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.187645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.187701 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.187833 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:51.188437 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.189110 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.190158 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.190508 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.190638 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.192201 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:51.192204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:51.192206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:51.192231 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:51.193938 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:51.194392 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:51.194395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:51.194396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:51.194478 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:51.195018 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:51.195022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:51.195024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:51.195276 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:51.195388 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.195748 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.195916 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 13:39:51.196200 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.186931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:37387 (local address 127.1.49.3:48513)
0926 13:39:51.187384 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.187422 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.187558 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.187901 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.187903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.188171 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:51.188304 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.188312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.189251 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.189254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.190020 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.190030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.190751 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.190755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.190832 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.191474 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:51.191486 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:51.192046 (+ 560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:51.194073 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:51.194076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:51.194079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:51.194291 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:51.194702 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:51.194710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:51.194716 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:51.194832 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:51.195417 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:51.195420 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:51.195541 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.195664 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.195865 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":138,"threads_started":1}
I20260926 13:39:51.197155 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:39:51 dist-test-slave-1wns krb5kdc[2291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:39:51 dist-test-slave-1wns krb5kdc[2291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:51.227345 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.217367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.1.49.62:37387)
0926 13:39:51.217529 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.217533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.217549 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:39:51.217597 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.217601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.217652 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.217740 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:51.218551 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.219181 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.220146 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.220388 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.220480 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.223406 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:51.223410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:51.223412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:51.223447 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:51.225238 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:51.225770 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:51.225773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:51.225774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:51.225872 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:51.226265 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:51.226268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:51.226270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:51.226485 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:51.226600 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.227074 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.227190 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 13:39:51.231206 2332 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:39:51.234390 2332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:39:51.251520 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.246102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.1.49.3:40553)
0926 13:39:51.246477 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.246488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.246507 (+ 19us) server_negotiation.cc:407] Connection header received
0926 13:39:51.246584 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.246589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.246747 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.247076 (+ 329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:51.247401 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.248614 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.249864 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.250651 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.250945 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.251077 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.251211 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.251333 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":126,"threads_started":1}
I20260926 13:39:51.251535 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.245679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:40553 (local address 127.0.0.1:52440)
0926 13:39:51.246167 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.246180 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.246250 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.246889 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.246891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.246908 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:51.247146 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.247155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.248741 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.248746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.249754 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.249764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.250808 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.250811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.250981 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.251028 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.251181 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.251368 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":101,"thread_start_us":98,"threads_started":1}
I20260926 13:39:51.252375 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.244869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:42193 (local address 127.0.0.1:33526)
0926 13:39:51.246052 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.246069 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.246178 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.246797 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.246803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.246825 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:51.247011 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.247019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.249369 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.249372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.250289 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.250297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.251569 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.251573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.251991 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.252056 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.252136 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.252217 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":73,"thread_start_us":106,"threads_started":1}
I20260926 13:39:51.252374 2836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.245231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:38567 (local address 127.0.0.1:44534)
0926 13:39:51.245807 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.245864 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.246024 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.246957 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.246960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.246974 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:51.247133 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.247139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.248397 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.248400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.249661 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.249670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.250789 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.250793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.251987 (+ 1194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.252107 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.252177 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.252231 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":333,"thread_start_us":108,"threads_started":1}
I20260926 13:39:51.252516 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.245716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.1.49.1:38567)
0926 13:39:51.246415 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.246422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.246444 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:39:51.246515 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.246519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.246664 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.246828 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:51.247267 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.248269 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.249796 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.250657 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.250782 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.250950 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.252238 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.252349 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":88,"threads_started":1}
I20260926 13:39:51.252861 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.245039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.1.49.2:42193)
0926 13:39:51.245811 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.245824 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.246288 (+ 464us) server_negotiation.cc:407] Connection header received
0926 13:39:51.246380 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.246386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.246621 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.246793 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:51.247150 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.249242 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.250415 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.251222 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.251328 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.251493 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.252645 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.252753 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":96,"threads_started":1}
I20260926 13:39:51.255012 2753 tablet_service.cc:1467] Processing CreateTablet for tablet 62d6fa589891400eb150041387471e2b (DEFAULT_TABLE table=test-table [id=f35583241a884a9bb03d029596606768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:51.255362 2473 tablet_service.cc:1467] Processing CreateTablet for tablet 62d6fa589891400eb150041387471e2b (DEFAULT_TABLE table=test-table [id=f35583241a884a9bb03d029596606768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:51.256306 2753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d6fa589891400eb150041387471e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:51.256534 2473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d6fa589891400eb150041387471e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:51.262740 2842 tablet_bootstrap.cc:492] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6: Bootstrap starting.
I20260926 13:39:51.263007 2613 tablet_service.cc:1467] Processing CreateTablet for tablet 62d6fa589891400eb150041387471e2b (DEFAULT_TABLE table=test-table [id=f35583241a884a9bb03d029596606768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:51.264235 2613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d6fa589891400eb150041387471e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:51.265025 2842 tablet_bootstrap.cc:654] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:51.265740 2842 log.cc:824] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6: Log is configured to *not* fsync() on all Append() calls
I20260926 13:39:51.267333 2843 tablet_bootstrap.cc:492] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131: Bootstrap starting.
I20260926 13:39:51.268059 2842 tablet_bootstrap.cc:492] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6: No bootstrap required, opened a new log
I20260926 13:39:51.268285 2842 ts_tablet_manager.cc:1397] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:39:51.269843 2843 tablet_bootstrap.cc:654] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:51.270835 2843 log.cc:824] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131: Log is configured to *not* fsync() on all Append() calls
I20260926 13:39:51.272498 2846 tablet_bootstrap.cc:492] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616: Bootstrap starting.
I20260926 13:39:51.273250 2843 tablet_bootstrap.cc:492] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131: No bootstrap required, opened a new log
I20260926 13:39:51.273120 2842 raft_consensus.cc:348] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } }
I20260926 13:39:51.273422 2842 raft_consensus.cc:374] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:51.273464 2843 ts_tablet_manager.cc:1397] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:39:51.273490 2842 raft_consensus.cc:729] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8425208696124113bfb0af38b7fc32f6, State: Initialized, Role: FOLLOWER
I20260926 13:39:51.273986 2842 consensus_queue.cc:260] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } }
I20260926 13:39:51.274848 2846 tablet_bootstrap.cc:654] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:51.275046 2842 ts_tablet_manager.cc:1428] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260926 13:39:51.275439 2539 heartbeater.cc:503] Master 127.1.49.62:37387 was elected leader, sending a full tablet report...
I20260926 13:39:51.275744 2846 log.cc:824] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616: Log is configured to *not* fsync() on all Append() calls
I20260926 13:39:51.277803 2846 tablet_bootstrap.cc:492] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616: No bootstrap required, opened a new log
I20260926 13:39:51.277711 2843 raft_consensus.cc:348] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } }
I20260926 13:39:51.278029 2843 raft_consensus.cc:374] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:51.278049 2846 ts_tablet_manager.cc:1397] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:39:51.278088 2843 raft_consensus.cc:729] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302f09efedd44c2482017c0dad16b131, State: Initialized, Role: FOLLOWER
I20260926 13:39:51.278580 2843 consensus_queue.cc:260] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } }
I20260926 13:39:51.279244 2819 heartbeater.cc:503] Master 127.1.49.62:37387 was elected leader, sending a full tablet report...
I20260926 13:39:51.279655 2843 ts_tablet_manager.cc:1428] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:39:51.281751 2846 raft_consensus.cc:348] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } }
I20260926 13:39:51.281987 2846 raft_consensus.cc:374] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:51.282051 2846 raft_consensus.cc:729] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63efd3ab1ee461391751867ba5a7616, State: Initialized, Role: FOLLOWER
I20260926 13:39:51.282464 2846 consensus_queue.cc:260] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } }
I20260926 13:39:51.283190 2679 heartbeater.cc:503] Master 127.1.49.62:37387 was elected leader, sending a full tablet report...
I20260926 13:39:51.283581 2846 ts_tablet_manager.cc:1428] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 13:39:51.288265 2847 raft_consensus.cc:482] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:39:51.288497 2847 raft_consensus.cc:504] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } }
I20260926 13:39:51.289637 2847 leader_election.cc:288] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d63efd3ab1ee461391751867ba5a7616 (127.1.49.2:42193), 302f09efedd44c2482017c0dad16b131 (127.1.49.3:40553)
I20260926 13:39:51.295466 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.290418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:40553 (local address 127.1.49.1:52815)
0926 13:39:51.290767 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.290781 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.290918 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.291237 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.291239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.291255 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:51.291386 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.291393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.292207 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.292213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.293642 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.293654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.294827 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.294833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.294952 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.294981 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.295112 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.295216 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":71,"thread_start_us":100,"threads_started":1}
I20260926 13:39:51.295670 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.290568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:52815 (local address 127.1.49.3:40553)
0926 13:39:51.290728 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.290732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.290896 (+ 164us) server_negotiation.cc:407] Connection header received
0926 13:39:51.291016 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.291019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.291127 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.291208 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:51.291502 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.292094 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.293820 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.294546 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.294678 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.294755 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.295178 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.295356 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 13:39:51.295894 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.290070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:39863 (local address 127.1.49.2:42193)
0926 13:39:51.290224 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.290231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.290481 (+ 250us) server_negotiation.cc:407] Connection header received
0926 13:39:51.290910 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.290914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.290979 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.291066 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:51.291383 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.292138 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.294173 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.295066 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.295188 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.295248 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.295622 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.295725 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 13:39:51.296491 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.289926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:42193 (local address 127.1.49.1:39863)
0926 13:39:51.290391 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.290406 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.290777 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.291084 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.291087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.291103 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:51.291263 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.291272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.292277 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.292280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.294005 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.294020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.295309 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.295313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.295448 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.295478 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.296068 (+ 590us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.296181 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":189,"thread_start_us":182,"threads_started":1}
I20260926 13:39:51.298292 2633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d6fa589891400eb150041387471e2b" candidate_uuid: "8425208696124113bfb0af38b7fc32f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63efd3ab1ee461391751867ba5a7616" is_pre_election: true
I20260926 13:39:51.298292 2773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d6fa589891400eb150041387471e2b" candidate_uuid: "8425208696124113bfb0af38b7fc32f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302f09efedd44c2482017c0dad16b131" is_pre_election: true
I20260926 13:39:51.298666 2633 raft_consensus.cc:2454] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8425208696124113bfb0af38b7fc32f6 in term 0.
I20260926 13:39:51.298666 2773 raft_consensus.cc:2454] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8425208696124113bfb0af38b7fc32f6 in term 0.
I20260926 13:39:51.299253 2427 leader_election.cc:302] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 302f09efedd44c2482017c0dad16b131, 8425208696124113bfb0af38b7fc32f6; no voters:
I20260926 13:39:51.299613 2847 raft_consensus.cc:2781] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:39:51.299715 2847 raft_consensus.cc:482] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:39:51.299762 2847 raft_consensus.cc:3037] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:51.301281 2847 raft_consensus.cc:504] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } }
I20260926 13:39:51.301749 2847 leader_election.cc:288] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [CANDIDATE]: Term 1 election: Requested vote from peers d63efd3ab1ee461391751867ba5a7616 (127.1.49.2:42193), 302f09efedd44c2482017c0dad16b131 (127.1.49.3:40553)
I20260926 13:39:51.302297 2633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d6fa589891400eb150041387471e2b" candidate_uuid: "8425208696124113bfb0af38b7fc32f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63efd3ab1ee461391751867ba5a7616"
I20260926 13:39:51.302367 2773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d6fa589891400eb150041387471e2b" candidate_uuid: "8425208696124113bfb0af38b7fc32f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302f09efedd44c2482017c0dad16b131"
I20260926 13:39:51.302466 2633 raft_consensus.cc:3037] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:51.302567 2773 raft_consensus.cc:3037] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:51.304318 2633 raft_consensus.cc:2454] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8425208696124113bfb0af38b7fc32f6 in term 1.
I20260926 13:39:51.304508 2773 raft_consensus.cc:2454] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8425208696124113bfb0af38b7fc32f6 in term 1.
I20260926 13:39:51.304834 2427 leader_election.cc:302] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8425208696124113bfb0af38b7fc32f6, d63efd3ab1ee461391751867ba5a7616; no voters:
I20260926 13:39:51.305090 2847 raft_consensus.cc:2781] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:39:51.305371 2847 raft_consensus.cc:686] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [term 1 LEADER]: Becoming Leader. State: Replica: 8425208696124113bfb0af38b7fc32f6, State: Running, Role: LEADER
I20260926 13:39:51.305859 2847 consensus_queue.cc:237] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } }
I20260926 13:39:51.309577 2336 catalog_manager.cc:5495] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8425208696124113bfb0af38b7fc32f6 (127.1.49.1). New cstate: current_term: 1 leader_uuid: "8425208696124113bfb0af38b7fc32f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } health_report { overall_health: HEALTHY } } }
I20260926 13:39:51.338873 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.334582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44538 (local address 127.1.49.1:38567)
0926 13:39:51.334715 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.334719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.334805 (+ 86us) server_negotiation.cc:407] Connection header received
0926 13:39:51.334994 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.334997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.335054 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.335141 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:39:51.335478 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.336102 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.337081 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.337348 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.337432 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.337805 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:39:51.338294 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:39:51.338386 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.338595 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.338662 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:39:51.345642 2773 raft_consensus.cc:1261] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131 [term 1 FOLLOWER]: Refusing update from remote peer 8425208696124113bfb0af38b7fc32f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:51.345717 2633 raft_consensus.cc:1261] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616 [term 1 FOLLOWER]: Refusing update from remote peer 8425208696124113bfb0af38b7fc32f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:51.346791 2853 consensus_queue.cc:1035] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:51.347028 2847 consensus_queue.cc:1035] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:51.355098 2856 mvcc.cc:205] Tried to move back new op lower bound from 7333601244543275008 to 7333601244392374272. Current Snapshot: MvccSnapshot[applied={T|T < 7333601244543275008}]
I20260926 13:39:51.355749 2857 mvcc.cc:205] Tried to move back new op lower bound from 7333601244543275008 to 7333601244392374272. Current Snapshot: MvccSnapshot[applied={T|T < 7333601244543275008}]
I20260926 13:39:51.356154 2858 mvcc.cc:205] Tried to move back new op lower bound from 7333601244543275008 to 7333601244392374272. Current Snapshot: MvccSnapshot[applied={T|T < 7333601244543275008}]
I20260926 13:39:51.368330 2336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:39:51.368551 2336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:39:51.371449 2336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea: Sending DeleteTablet for 3 replicas of tablet 62d6fa589891400eb150041387471e2b
I20260926 13:39:51.372609 2753 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d6fa589891400eb150041387471e2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:51 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:52440
I20260926 13:39:51.372690 2473 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d6fa589891400eb150041387471e2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:51 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:44534
I20260926 13:39:51.372753 2613 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d6fa589891400eb150041387471e2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:51 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:33526
I20260926 13:39:51.373224 2867 tablet_replica.cc:330] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6: stopping tablet replica
I20260926 13:39:51.373472 2866 tablet_replica.cc:330] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131: stopping tablet replica
I20260926 13:39:51.373446 2336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 13:39:51.373726 2867 raft_consensus.cc:2229] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:39:51.373926 2868 tablet_replica.cc:330] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616: stopping tablet replica
W20260926 13:39:51.373967 2336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:39:51.374200 2867 raft_consensus.cc:2258] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:51.374608 2868 raft_consensus.cc:2229] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:39:51.374858 2866 raft_consensus.cc:2229] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:39:51.375273 2866 raft_consensus.cc:2258] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:51.375617 2868 raft_consensus.cc:2258] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:51.377563 2868 ts_tablet_manager.cc:1905] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:51.380311 2753 tablet_service.cc:1467] Processing CreateTablet for tablet 28e9499049964a62964cadbff26e09ac (DEFAULT_TABLE table=test-table [id=e1934b7111444b559a9fd0a2a31638cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:51.380298 2473 tablet_service.cc:1467] Processing CreateTablet for tablet 28e9499049964a62964cadbff26e09ac (DEFAULT_TABLE table=test-table [id=e1934b7111444b559a9fd0a2a31638cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:51.380604 2753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e9499049964a62964cadbff26e09ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:51.380625 2473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e9499049964a62964cadbff26e09ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:51.380764 2868 ts_tablet_manager.cc:1918] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:39:51.380911 2868 log.cc:1196] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/62d6fa589891400eb150041387471e2b
I20260926 13:39:51.381312 2868 ts_tablet_manager.cc:1939] T 62d6fa589891400eb150041387471e2b P d63efd3ab1ee461391751867ba5a7616: Deleting consensus metadata
I20260926 13:39:51.383076 2613 tablet_service.cc:1467] Processing CreateTablet for tablet 28e9499049964a62964cadbff26e09ac (DEFAULT_TABLE table=test-table [id=e1934b7111444b559a9fd0a2a31638cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:51.383164 2843 tablet_bootstrap.cc:492] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131: Bootstrap starting.
I20260926 13:39:51.383327 2613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e9499049964a62964cadbff26e09ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:51.384146 2323 catalog_manager.cc:4841] TS d63efd3ab1ee461391751867ba5a7616 (127.1.49.2:42193): tablet 62d6fa589891400eb150041387471e2b (table test-table [id=f35583241a884a9bb03d029596606768]) successfully deleted
I20260926 13:39:51.384807 2843 tablet_bootstrap.cc:654] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:51.385974 2842 tablet_bootstrap.cc:492] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6: Bootstrap starting.
I20260926 13:39:51.386268 2846 tablet_bootstrap.cc:492] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616: Bootstrap starting.
I20260926 13:39:51.387023 2842 tablet_bootstrap.cc:654] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:51.387187 2846 tablet_bootstrap.cc:654] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:51.388957 2846 tablet_bootstrap.cc:492] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616: No bootstrap required, opened a new log
I20260926 13:39:51.389066 2846 ts_tablet_manager.cc:1397] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 13:39:51.389549 2846 raft_consensus.cc:348] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.389681 2846 raft_consensus.cc:374] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:51.389726 2846 raft_consensus.cc:729] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63efd3ab1ee461391751867ba5a7616, State: Initialized, Role: FOLLOWER
I20260926 13:39:51.389854 2846 consensus_queue.cc:260] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.389964 2866 ts_tablet_manager.cc:1905] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:51.390136 2846 ts_tablet_manager.cc:1428] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:39:51.390298 2867 ts_tablet_manager.cc:1905] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:51.391067 2843 tablet_bootstrap.cc:492] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131: No bootstrap required, opened a new log
I20260926 13:39:51.391171 2843 ts_tablet_manager.cc:1397] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260926 13:39:51.391671 2843 raft_consensus.cc:348] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.391847 2843 raft_consensus.cc:374] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:51.391897 2843 raft_consensus.cc:729] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302f09efedd44c2482017c0dad16b131, State: Initialized, Role: FOLLOWER
I20260926 13:39:51.392040 2843 consensus_queue.cc:260] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.392338 2843 ts_tablet_manager.cc:1428] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260926 13:39:51.393855 2867 ts_tablet_manager.cc:1918] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:39:51.394030 2867 log.cc:1196] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/wal/wals/62d6fa589891400eb150041387471e2b
I20260926 13:39:51.394470 2867 ts_tablet_manager.cc:1939] T 62d6fa589891400eb150041387471e2b P 8425208696124113bfb0af38b7fc32f6: Deleting consensus metadata
I20260926 13:39:51.395345 2842 tablet_bootstrap.cc:492] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6: No bootstrap required, opened a new log
I20260926 13:39:51.395447 2842 ts_tablet_manager.cc:1397] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260926 13:39:51.396028 2321 catalog_manager.cc:4841] TS 8425208696124113bfb0af38b7fc32f6 (127.1.49.1:38567): tablet 62d6fa589891400eb150041387471e2b (table test-table [id=f35583241a884a9bb03d029596606768]) successfully deleted
I20260926 13:39:51.395999 2842 raft_consensus.cc:348] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.396113 2842 raft_consensus.cc:374] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:51.396144 2842 raft_consensus.cc:729] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8425208696124113bfb0af38b7fc32f6, State: Initialized, Role: FOLLOWER
I20260926 13:39:51.396272 2842 consensus_queue.cc:260] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.396555 2842 ts_tablet_manager.cc:1428] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:39:51.398802 2866 ts_tablet_manager.cc:1918] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:39:51.398983 2866 log.cc:1196] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/62d6fa589891400eb150041387471e2b
I20260926 13:39:51.399400 2866 ts_tablet_manager.cc:1939] T 62d6fa589891400eb150041387471e2b P 302f09efedd44c2482017c0dad16b131: Deleting consensus metadata
I20260926 13:39:51.400722 2323 catalog_manager.cc:4841] TS 302f09efedd44c2482017c0dad16b131 (127.1.49.3:40553): tablet 62d6fa589891400eb150041387471e2b (table test-table [id=f35583241a884a9bb03d029596606768]) successfully deleted
W20260926 13:39:51.430476 2820 tablet.cc:2369] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:39:51.436383 2680 tablet.cc:2369] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:39:51.442126 2540 tablet.cc:2369] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:39:51.596873 2849 raft_consensus.cc:482] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:39:51.597106 2849 raft_consensus.cc:504] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.598456 2849 leader_election.cc:288] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8425208696124113bfb0af38b7fc32f6 (127.1.49.1:38567), d63efd3ab1ee461391751867ba5a7616 (127.1.49.2:42193)
I20260926 13:39:51.602293 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.598797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:50757 (local address 127.1.49.1:38567)
0926 13:39:51.598958 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.598966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.599082 (+ 116us) server_negotiation.cc:407] Connection header received
0926 13:39:51.599252 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.599256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.599302 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.599388 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:51.599644 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.600217 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.601253 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.601673 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.601788 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.601860 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.602084 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.602158 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 13:39:51.602293 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.598690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:38567 (local address 127.1.49.3:50757)
0926 13:39:51.598958 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.598977 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.599120 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.599399 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.599401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.599417 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:51.599537 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.599542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.600325 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.600332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.601123 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.601133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.601793 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.601796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.601901 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.601956 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.602062 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.602145 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":18}
I20260926 13:39:51.602429 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.598938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:38135 (local address 127.1.49.2:42193)
0926 13:39:51.599072 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.599076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.599267 (+ 191us) server_negotiation.cc:407] Connection header received
0926 13:39:51.599431 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.599434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.599484 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.599577 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:51.599863 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.600346 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.601391 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.601789 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.601908 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.601974 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.602200 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.602277 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 13:39:51.602495 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.598817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:42193 (local address 127.1.49.3:38135)
0926 13:39:51.599176 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.599187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.599312 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.599588 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.599591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.599606 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:51.599728 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.599733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.600454 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.600460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.601266 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.601274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.601909 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.601912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.602035 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.602062 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.602153 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.602236 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260926 13:39:51.602931 2493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e9499049964a62964cadbff26e09ac" candidate_uuid: "302f09efedd44c2482017c0dad16b131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8425208696124113bfb0af38b7fc32f6" is_pre_election: true
I20260926 13:39:51.603042 2633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e9499049964a62964cadbff26e09ac" candidate_uuid: "302f09efedd44c2482017c0dad16b131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63efd3ab1ee461391751867ba5a7616" is_pre_election: true
I20260926 13:39:51.603201 2633 raft_consensus.cc:2454] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 302f09efedd44c2482017c0dad16b131 in term 0.
I20260926 13:39:51.603277 2493 raft_consensus.cc:2454] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 302f09efedd44c2482017c0dad16b131 in term 0.
I20260926 13:39:51.603637 2707 leader_election.cc:302] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 302f09efedd44c2482017c0dad16b131, d63efd3ab1ee461391751867ba5a7616; no voters:
I20260926 13:39:51.603994 2849 raft_consensus.cc:2781] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:39:51.604068 2849 raft_consensus.cc:482] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:39:51.604107 2849 raft_consensus.cc:3037] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:51.605234 2849 raft_consensus.cc:504] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.605660 2849 leader_election.cc:288] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [CANDIDATE]: Term 1 election: Requested vote from peers 8425208696124113bfb0af38b7fc32f6 (127.1.49.1:38567), d63efd3ab1ee461391751867ba5a7616 (127.1.49.2:42193)
I20260926 13:39:51.606079 2633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e9499049964a62964cadbff26e09ac" candidate_uuid: "302f09efedd44c2482017c0dad16b131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d63efd3ab1ee461391751867ba5a7616"
I20260926 13:39:51.606232 2633 raft_consensus.cc:3037] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:51.606240 2493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e9499049964a62964cadbff26e09ac" candidate_uuid: "302f09efedd44c2482017c0dad16b131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8425208696124113bfb0af38b7fc32f6"
I20260926 13:39:51.606402 2493 raft_consensus.cc:3037] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:51.607367 2633 raft_consensus.cc:2454] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 302f09efedd44c2482017c0dad16b131 in term 1.
I20260926 13:39:51.607496 2493 raft_consensus.cc:2454] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 302f09efedd44c2482017c0dad16b131 in term 1.
I20260926 13:39:51.607784 2707 leader_election.cc:302] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 302f09efedd44c2482017c0dad16b131, d63efd3ab1ee461391751867ba5a7616; no voters:
I20260926 13:39:51.607999 2849 raft_consensus.cc:2781] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:39:51.608242 2849 raft_consensus.cc:686] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [term 1 LEADER]: Becoming Leader. State: Replica: 302f09efedd44c2482017c0dad16b131, State: Running, Role: LEADER
I20260926 13:39:51.608533 2849 consensus_queue.cc:237] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.611337 2336 catalog_manager.cc:5495] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 reported cstate change: term changed from 0 to 1, leader changed from <none> to 302f09efedd44c2482017c0dad16b131 (127.1.49.3). New cstate: current_term: 1 leader_uuid: "302f09efedd44c2482017c0dad16b131" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } health_report { overall_health: HEALTHY } } }
Sep 26 13:39:51 dist-test-slave-1wns krb5kdc[2291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429990, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:51.668403 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.658020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.1.49.62:37387)
0926 13:39:51.658195 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.658199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.658500 (+ 301us) server_negotiation.cc:407] Connection header received
0926 13:39:51.658552 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.658554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.658647 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.658749 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:51.659541 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.660330 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.661232 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.661448 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.661541 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.664264 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:51.664268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:51.664270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:51.664294 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:51.665993 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:51.666561 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:51.666564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:51.666566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:51.666738 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:51.667168 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:51.667171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:51.667172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:51.667457 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:51.667553 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.668120 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.668275 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":71}
I20260926 13:39:51.668406 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.657960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:37387 (local address 127.0.0.1:43956)
0926 13:39:51.658299 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.658318 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.658452 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.658801 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.658804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.659273 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:51.659375 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.659381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.660455 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.660458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.661088 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.661098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.661592 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.661596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.661675 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.662329 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:51.662342 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:51.664068 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:51.666133 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:51.666153 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:51.666159 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:51.666443 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:51.666876 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:51.666879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:51.666880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:51.667031 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:51.667620 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:51.667627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:51.667781 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.668023 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.668203 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":142,"threads_started":1}
I20260926 13:39:51.673610 2336 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:43956:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 13:39:51.674459 2336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:39:51.680768 2753 tablet_service.cc:1467] Processing CreateTablet for tablet f8ae50d4746343c4bdabc20701b96c6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63a49bda85f044e38a42ddf282ccd4e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:39:51.680757 2613 tablet_service.cc:1467] Processing CreateTablet for tablet f8ae50d4746343c4bdabc20701b96c6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63a49bda85f044e38a42ddf282ccd4e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:39:51.680757 2473 tablet_service.cc:1467] Processing CreateTablet for tablet f8ae50d4746343c4bdabc20701b96c6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63a49bda85f044e38a42ddf282ccd4e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:39:51.681195 2613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ae50d4746343c4bdabc20701b96c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:51.681195 2473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ae50d4746343c4bdabc20701b96c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:51.681201 2753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ae50d4746343c4bdabc20701b96c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:51.684227 2843 tablet_bootstrap.cc:492] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131: Bootstrap starting.
I20260926 13:39:51.684830 2842 tablet_bootstrap.cc:492] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6: Bootstrap starting.
I20260926 13:39:51.685372 2843 tablet_bootstrap.cc:654] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:51.685564 2842 tablet_bootstrap.cc:654] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:51.686127 2846 tablet_bootstrap.cc:492] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616: Bootstrap starting.
I20260926 13:39:51.686887 2842 tablet_bootstrap.cc:492] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6: No bootstrap required, opened a new log
I20260926 13:39:51.686983 2842 ts_tablet_manager.cc:1397] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:39:51.687054 2843 tablet_bootstrap.cc:492] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131: No bootstrap required, opened a new log
I20260926 13:39:51.687146 2843 ts_tablet_manager.cc:1397] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 13:39:51.687179 2846 tablet_bootstrap.cc:654] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:51.687494 2842 raft_consensus.cc:348] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.687646 2842 raft_consensus.cc:374] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:51.687580 2843 raft_consensus.cc:348] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.687683 2842 raft_consensus.cc:729] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8425208696124113bfb0af38b7fc32f6, State: Initialized, Role: FOLLOWER
I20260926 13:39:51.687690 2843 raft_consensus.cc:374] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:51.687729 2843 raft_consensus.cc:729] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302f09efedd44c2482017c0dad16b131, State: Initialized, Role: FOLLOWER
I20260926 13:39:51.687887 2843 consensus_queue.cc:260] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.688218 2843 ts_tablet_manager.cc:1428] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 13:39:51.688269 2842 consensus_queue.cc:260] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.688449 2846 tablet_bootstrap.cc:492] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616: No bootstrap required, opened a new log
I20260926 13:39:51.688544 2846 ts_tablet_manager.cc:1397] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:39:51.688470 2849 tablet_replica.cc:439] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } }
I20260926 13:39:51.688619 2849 tablet_replica.cc:442] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:39:51.688536 2861 tablet_replica.cc:439] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } }
I20260926 13:39:51.688668 2861 tablet_replica.cc:442] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:39:51.688890 2842 ts_tablet_manager.cc:1428] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:39:51.689041 2846 raft_consensus.cc:348] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.689162 2846 raft_consensus.cc:374] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:51.689198 2846 raft_consensus.cc:729] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63efd3ab1ee461391751867ba5a7616, State: Initialized, Role: FOLLOWER
I20260926 13:39:51.689316 2846 consensus_queue.cc:260] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.689540 2846 ts_tablet_manager.cc:1428] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:39:51.689756 2850 tablet_replica.cc:439] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } }
I20260926 13:39:51.689889 2850 tablet_replica.cc:442] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:39:51.778330 2850 raft_consensus.cc:482] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:39:51.778481 2850 raft_consensus.cc:504] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.779698 2850 leader_election.cc:288] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8425208696124113bfb0af38b7fc32f6 (127.1.49.1:38567), 302f09efedd44c2482017c0dad16b131 (127.1.49.3:40553)
I20260926 13:39:51.784161 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.780312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:55693 (local address 127.1.49.1:38567)
0926 13:39:51.780480 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.780487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.780784 (+ 297us) server_negotiation.cc:407] Connection header received
0926 13:39:51.780904 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.780907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.780958 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.781046 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:51.781318 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.781796 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.783034 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.783467 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.783605 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.783697 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.783922 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.784027 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:39:51.784161 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.780156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:56143 (local address 127.1.49.3:40553)
0926 13:39:51.780324 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.780329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.780610 (+ 281us) server_negotiation.cc:407] Connection header received
0926 13:39:51.780739 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.780744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.780795 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.780873 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:51.781185 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.781760 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.783054 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.783464 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.783601 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.783673 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.783957 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.784051 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 13:39:51.784179 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.780051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:38567 (local address 127.1.49.2:55693)
0926 13:39:51.780645 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.780680 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.780781 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.781052 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.781054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.781068 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:51.781202 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.781210 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.781936 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.781938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.782888 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.782897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.783618 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.783621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.783758 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.783780 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.783890 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.783973 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":103,"spinlock_wait_cycles":14592,"thread_start_us":112,"threads_started":1}
I20260926 13:39:51.784586 2885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.780051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:40553 (local address 127.1.49.2:56143)
0926 13:39:51.780511 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.780523 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.780630 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.780904 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.780907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.780924 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:51.781062 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.781070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.781882 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.781888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.782908 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.782917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.783618 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.783621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.783773 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.783790 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.784320 (+ 530us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.784409 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":117,"thread_start_us":106,"threads_started":1}
I20260926 13:39:51.784749 2493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ae50d4746343c4bdabc20701b96c6b" candidate_uuid: "d63efd3ab1ee461391751867ba5a7616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8425208696124113bfb0af38b7fc32f6" is_pre_election: true
I20260926 13:39:51.784942 2493 raft_consensus.cc:2454] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d63efd3ab1ee461391751867ba5a7616 in term 0.
I20260926 13:39:51.785045 2773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ae50d4746343c4bdabc20701b96c6b" candidate_uuid: "d63efd3ab1ee461391751867ba5a7616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302f09efedd44c2482017c0dad16b131" is_pre_election: true
I20260926 13:39:51.785202 2773 raft_consensus.cc:2454] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d63efd3ab1ee461391751867ba5a7616 in term 0.
I20260926 13:39:51.785373 2565 leader_election.cc:302] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8425208696124113bfb0af38b7fc32f6, d63efd3ab1ee461391751867ba5a7616; no voters:
I20260926 13:39:51.785670 2850 raft_consensus.cc:2781] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:39:51.785732 2850 raft_consensus.cc:482] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:39:51.785770 2850 raft_consensus.cc:3037] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:51.787287 2850 raft_consensus.cc:504] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.787747 2850 leader_election.cc:288] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [CANDIDATE]: Term 1 election: Requested vote from peers 8425208696124113bfb0af38b7fc32f6 (127.1.49.1:38567), 302f09efedd44c2482017c0dad16b131 (127.1.49.3:40553)
I20260926 13:39:51.788244 2493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ae50d4746343c4bdabc20701b96c6b" candidate_uuid: "d63efd3ab1ee461391751867ba5a7616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8425208696124113bfb0af38b7fc32f6"
I20260926 13:39:51.788372 2493 raft_consensus.cc:3037] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:51.788290 2773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ae50d4746343c4bdabc20701b96c6b" candidate_uuid: "d63efd3ab1ee461391751867ba5a7616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302f09efedd44c2482017c0dad16b131"
I20260926 13:39:51.788663 2773 raft_consensus.cc:3037] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:51.789469 2493 raft_consensus.cc:2454] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d63efd3ab1ee461391751867ba5a7616 in term 1.
I20260926 13:39:51.789700 2773 raft_consensus.cc:2454] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d63efd3ab1ee461391751867ba5a7616 in term 1.
I20260926 13:39:51.789871 2565 leader_election.cc:302] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8425208696124113bfb0af38b7fc32f6, d63efd3ab1ee461391751867ba5a7616; no voters:
I20260926 13:39:51.790068 2850 raft_consensus.cc:2781] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:39:51.790292 2850 raft_consensus.cc:686] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [term 1 LEADER]: Becoming Leader. State: Replica: d63efd3ab1ee461391751867ba5a7616, State: Running, Role: LEADER
I20260926 13:39:51.790627 2850 consensus_queue.cc:237] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } }
I20260926 13:39:51.792203 2886 tablet_replica.cc:439] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616: TxnStatusTablet state changed. Reason: New leader d63efd3ab1ee461391751867ba5a7616. Latest consensus state: current_term: 1 leader_uuid: "d63efd3ab1ee461391751867ba5a7616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } }
I20260926 13:39:51.792330 2886 tablet_replica.cc:442] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:39:51.792615 2888 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 13:39:51.792798 2888 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 13:39:51.793411 2335 catalog_manager.cc:5495] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 reported cstate change: term changed from 0 to 1, leader changed from <none> to d63efd3ab1ee461391751867ba5a7616 (127.1.49.2). New cstate: current_term: 1 leader_uuid: "d63efd3ab1ee461391751867ba5a7616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260926 13:39:51.820312 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.815904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.1.49.2:42193)
0926 13:39:51.816071 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.816075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.816098 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:39:51.816321 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.816324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.816383 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.816478 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:39:51.816791 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.817319 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.818599 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.818846 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.818940 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.819351 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:39:51.819917 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:39:51.820018 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.820135 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.820191 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 13:39:51.820415 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.815745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:42193 (local address 127.0.0.1:33536)
0926 13:39:51.815975 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.815993 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.816115 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.816515 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.816518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.816529 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:39:51.816649 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.816654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.817434 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.817437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.818419 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.818431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.818978 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.818982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.819080 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.819146 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:39:51.820014 (+ 868us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:39:51.820034 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.820115 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.820257 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 13:39:51.823709 2773 raft_consensus.cc:1261] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131 [term 1 FOLLOWER]: Refusing update from remote peer d63efd3ab1ee461391751867ba5a7616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:51.824061 2493 raft_consensus.cc:1261] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6 [term 1 FOLLOWER]: Refusing update from remote peer d63efd3ab1ee461391751867ba5a7616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:51.824369 2886 consensus_queue.cc:1035] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:51.824680 2850 consensus_queue.cc:1035] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:51.825943 2849 tablet_replica.cc:439] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131: TxnStatusTablet state changed. Reason: New leader d63efd3ab1ee461391751867ba5a7616. Latest consensus state: current_term: 1 leader_uuid: "d63efd3ab1ee461391751867ba5a7616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } }
I20260926 13:39:51.826071 2849 tablet_replica.cc:442] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:39:51.827718 2861 tablet_replica.cc:439] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6: TxnStatusTablet state changed. Reason: New leader d63efd3ab1ee461391751867ba5a7616. Latest consensus state: current_term: 1 leader_uuid: "d63efd3ab1ee461391751867ba5a7616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } }
I20260926 13:39:51.827895 2861 tablet_replica.cc:442] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:39:51.830524 2886 tablet_replica.cc:439] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d63efd3ab1ee461391751867ba5a7616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } }
I20260926 13:39:51.830672 2893 tablet_replica.cc:439] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d63efd3ab1ee461391751867ba5a7616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } }
I20260926 13:39:51.830753 2893 tablet_replica.cc:442] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:39:51.831662 2849 tablet_replica.cc:439] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d63efd3ab1ee461391751867ba5a7616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } }
I20260926 13:39:51.831784 2849 tablet_replica.cc:442] T f8ae50d4746343c4bdabc20701b96c6b P 302f09efedd44c2482017c0dad16b131: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:39:51.832311 2861 tablet_replica.cc:439] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d63efd3ab1ee461391751867ba5a7616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 } } peers { permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 } } peers { permanent_uuid: "302f09efedd44c2482017c0dad16b131" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40553 } } }
I20260926 13:39:51.832430 2861 tablet_replica.cc:442] T f8ae50d4746343c4bdabc20701b96c6b P 8425208696124113bfb0af38b7fc32f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:39:51.830684 2886 tablet_replica.cc:442] T f8ae50d4746343c4bdabc20701b96c6b P d63efd3ab1ee461391751867ba5a7616: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:39:51.841256 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.837271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.1.49.3:40553)
0926 13:39:51.837384 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.837391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.837429 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:39:51.837665 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.837669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.837737 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.837861 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:39:51.838092 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.838599 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.839513 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.839880 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.839988 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.840307 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:39:51.840858 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:39:51.840986 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.841048 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.841112 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:39:51.854586 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.850171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:54305 (local address 127.1.49.2:42193)
0926 13:39:51.850323 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.850329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.850619 (+ 290us) server_negotiation.cc:407] Connection header received
0926 13:39:51.850775 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.850778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.850832 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.850920 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:39:51.851217 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.852071 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.853341 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.853568 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.853752 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.854089 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:39:51.854172 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:39:51.854239 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.854394 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.854454 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:39:51.854607 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.850126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:42193 (local address 127.1.49.3:54305)
0926 13:39:51.850524 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.850540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.850667 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.850946 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.850948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.850955 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:39:51.851090 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.851096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.852217 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.852220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.853229 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.853240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.853788 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.853792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.853894 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.853957 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:39:51.854264 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:39:51.854270 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.854338 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.854419 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":160,"threads_started":1}
I20260926 13:39:51.862900 2493 raft_consensus.cc:1261] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6 [term 1 FOLLOWER]: Refusing update from remote peer 302f09efedd44c2482017c0dad16b131: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:51.862900 2633 raft_consensus.cc:1261] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616 [term 1 FOLLOWER]: Refusing update from remote peer 302f09efedd44c2482017c0dad16b131: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:51.863415 2873 consensus_queue.cc:1035] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8425208696124113bfb0af38b7fc32f6" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:51.863723 2849 consensus_queue.cc:1035] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d63efd3ab1ee461391751867ba5a7616" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 42193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:51.893590 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.888992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:47877 (local address 127.1.49.3:40553)
0926 13:39:51.889247 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.889252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.889323 (+ 71us) server_negotiation.cc:407] Connection header received
0926 13:39:51.889525 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.889528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.889584 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.889670 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:39:51.889969 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.890760 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.892065 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.892392 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.892521 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.893009 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:39:51.893143 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:39:51.893218 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.893370 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.893433 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260926 13:39:51.894249 2902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.888911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:40553 (local address 127.1.49.2:47877)
0926 13:39:51.889248 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:51.889264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:51.889378 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:51.889693 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:51.889696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:51.889703 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:39:51.889841 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.889847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.890906 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.890909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.891929 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:51.891940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.892734 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.892737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:51.892813 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.892865 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:39:51.893244 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:39:51.893252 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 13:39:51.893991 (+ 739us) client_negotiation.cc:240] Negotiation successful
0926 13:39:51.894082 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":114,"threads_started":1}
I20260926 13:39:51.919569 2335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:39:51.919770 2335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:39:51.922411 2335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac6647ea3eaa4561b1d0b8ce208cf3ea: Sending DeleteTablet for 3 replicas of tablet 28e9499049964a62964cadbff26e09ac
I20260926 13:39:51.923141 2473 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e9499049964a62964cadbff26e09ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:51 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:44534
I20260926 13:39:51.923550 2613 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e9499049964a62964cadbff26e09ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:51 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:33526
I20260926 13:39:51.923904 2906 tablet_replica.cc:330] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6: stopping tablet replica
I20260926 13:39:51.923529 2753 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e9499049964a62964cadbff26e09ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:51 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:52440
I20260926 13:39:51.924161 2906 raft_consensus.cc:2229] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:39:51.924350 2906 raft_consensus.cc:2258] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:51.925333 2906 ts_tablet_manager.cc:1905] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:51.925781 2908 tablet_replica.cc:330] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131: stopping tablet replica
I20260926 13:39:51.925931 2908 raft_consensus.cc:2229] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:39:51.926275 2908 raft_consensus.cc:2258] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:51.927830 2908 ts_tablet_manager.cc:1905] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:51.929279 2906 ts_tablet_manager.cc:1918] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:39:51.929423 2906 log.cc:1196] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-0/wal/wals/28e9499049964a62964cadbff26e09ac
I20260926 13:39:51.929773 2906 ts_tablet_manager.cc:1939] T 28e9499049964a62964cadbff26e09ac P 8425208696124113bfb0af38b7fc32f6: Deleting consensus metadata
I20260926 13:39:51.930886 2321 catalog_manager.cc:4841] TS 8425208696124113bfb0af38b7fc32f6 (127.1.49.1:38567): tablet 28e9499049964a62964cadbff26e09ac (table test-table [id=e1934b7111444b559a9fd0a2a31638cf]) successfully deleted
I20260926 13:39:51.931100 2908 ts_tablet_manager.cc:1918] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:39:51.931206 2908 log.cc:1196] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/28e9499049964a62964cadbff26e09ac
I20260926 13:39:51.931474 2908 ts_tablet_manager.cc:1939] T 28e9499049964a62964cadbff26e09ac P 302f09efedd44c2482017c0dad16b131: Deleting consensus metadata
I20260926 13:39:51.932242 2323 catalog_manager.cc:4841] TS 302f09efedd44c2482017c0dad16b131 (127.1.49.3:40553): tablet 28e9499049964a62964cadbff26e09ac (table test-table [id=e1934b7111444b559a9fd0a2a31638cf]) successfully deleted
I20260926 13:39:51.932884 2907 tablet_replica.cc:330] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616: stopping tablet replica
I20260926 13:39:51.933022 2907 raft_consensus.cc:2229] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:39:51.933152 2907 raft_consensus.cc:2258] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:51.933990 2907 ts_tablet_manager.cc:1905] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:51.937391 2907 ts_tablet_manager.cc:1918] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:39:51.937472 2907 log.cc:1196] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/28e9499049964a62964cadbff26e09ac
I20260926 13:39:51.937734 2907 ts_tablet_manager.cc:1939] T 28e9499049964a62964cadbff26e09ac P d63efd3ab1ee461391751867ba5a7616: Deleting consensus metadata
I20260926 13:39:51.940092 2323 catalog_manager.cc:4841] TS d63efd3ab1ee461391751867ba5a7616 (127.1.49.2:42193): tablet 28e9499049964a62964cadbff26e09ac (table test-table [id=e1934b7111444b559a9fd0a2a31638cf]) successfully deleted
Sep 26 13:39:51 dist-test-slave-1wns krb5kdc[2291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429991, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.49.1@KRBTEST.COM
I20260926 13:39:51.951221 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.926790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44540 (local address 127.1.49.1:38567)
0926 13:39:51.926908 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.926912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.932570 (+ 5658us) server_negotiation.cc:407] Connection header received
0926 13:39:51.932717 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.932721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.932775 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.932866 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:51.939065 (+ 6199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.939977 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.941320 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.941693 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.941774 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.946614 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:51.946633 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:51.946651 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:51.946689 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:51.948913 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:51.949555 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:51.949563 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:51.949565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:51.949671 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:51.949984 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:51.949988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:51.949989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:51.950331 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:51.950522 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.950899 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.951037 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 13:39:51.952502 2529 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44540
I20260926 13:39:51.965797 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:51.955325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.1.49.62:37387)
0926 13:39:51.955504 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:51.955509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:51.956050 (+ 541us) server_negotiation.cc:407] Connection header received
0926 13:39:51.956321 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:51.956325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:51.956379 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:51.956433 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:51.957394 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.958225 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.959438 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:51.959786 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:51.959918 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:51.961854 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:51.961858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:51.961861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:51.961897 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:51.963850 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:51.964375 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:51.964379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:51.964381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:51.964474 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:51.964774 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:51.964777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:51.964779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:51.965003 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:51.965123 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:51.965488 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 13:39:51.965625 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 13:39:51.966729 2335 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43972
I20260926 13:39:51.968585 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 2408
I20260926 13:39:51.978233 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 2548
I20260926 13:39:51.985209 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 2688
I20260926 13:39:51.991545 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 2304
2026-09-26T13:39:51Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3037 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:39:52 dist-test-slave-1wns krb5kdc[2915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:39:52 dist-test-slave-1wns krb5kdc[2915](info): set up 2 sockets
Sep 26 13:39:52 dist-test-slave-1wns krb5kdc[2915](info): commencing operation
krb5kdc: starting...
W20260926 13:39:53.075783 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:39:53 dist-test-slave-1wns krb5kdc[2915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:39:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:39:53.215988 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:33571
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:33571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:53.348526 2928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:53.348961 2928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:53.349004 2928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:53.353374 2928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:39:53.353456 2928 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 13:39:53.353474 2928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:53.353492 2928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:53.353510 2928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:39:53.353526 2928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:39:53.359354 2928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:33571
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:33571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:53.360627 2928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:53.361801 2928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:53.371031 2936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:53.371101 2928 server_base.cc:1034] running on GCE node
W20260926 13:39:53.371054 2934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:53.371040 2933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:53.371920 2928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:53.373179 2928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:53.374356 2928 hybrid_clock.cc:648] HybridClock initialized: now 1790429993374298 us; error 74 us; skew 500 ppm
Sep 26 13:39:53 dist-test-slave-1wns krb5kdc[2915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429993, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:53.384253 2928 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:39:53.385545 2928 webserver.cc:460] Webserver started at http://127.1.49.62:41743/ using document root <none> and password file <none>
I20260926 13:39:53.386305 2928 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:53.386428 2928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:53.386718 2928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:53.389318 2928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "241452488bed4fb1ad4969113583b95a"
format_stamp: "Formatted at 2026-09-26 13:39:53 on dist-test-slave-1wns"
server_key: "17c2a5177c2a00c519375bda1b2ebdeb"
server_key_iv: "bdd3e6c545e38ef3ed5d9b1aa57626fe"
server_key_version: "encryptionkey@0"
I20260926 13:39:53.390015 2928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "241452488bed4fb1ad4969113583b95a"
format_stamp: "Formatted at 2026-09-26 13:39:53 on dist-test-slave-1wns"
server_key: "17c2a5177c2a00c519375bda1b2ebdeb"
server_key_iv: "bdd3e6c545e38ef3ed5d9b1aa57626fe"
server_key_version: "encryptionkey@0"
I20260926 13:39:53.394490 2928 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:39:53.397501 2943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:53.399022 2928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:39:53.399206 2928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "241452488bed4fb1ad4969113583b95a"
format_stamp: "Formatted at 2026-09-26 13:39:53 on dist-test-slave-1wns"
server_key: "17c2a5177c2a00c519375bda1b2ebdeb"
server_key_iv: "bdd3e6c545e38ef3ed5d9b1aa57626fe"
server_key_version: "encryptionkey@0"
I20260926 13:39:53.399349 2928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:53.424803 2928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:53.428925 2928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:53.429164 2928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:53.438444 2928 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:33571
I20260926 13:39:53.438444 3005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:33571 every 8 connection(s)
I20260926 13:39:53.439574 2928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:39:53.442740 3006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:53.444502 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 2928
I20260926 13:39:53.444628 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:39:53.444898 1220 external_mini_cluster.cc:1428] Setting key 3de88f3d56002aef331d71f0310497c1
I20260926 13:39:53.449440 3006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a: Bootstrap starting.
I20260926 13:39:53.451792 3006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:53.452730 3006 log.cc:824] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a: Log is configured to *not* fsync() on all Append() calls
Sep 26 13:39:53 dist-test-slave-1wns krb5kdc[2915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:53.455299 3006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a: No bootstrap required, opened a new log
I20260926 13:39:53.458806 3006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241452488bed4fb1ad4969113583b95a" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33571 } }
I20260926 13:39:53.459097 3006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:53.459188 3006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241452488bed4fb1ad4969113583b95a, State: Initialized, Role: FOLLOWER
I20260926 13:39:53.459743 3006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241452488bed4fb1ad4969113583b95a" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33571 } }
I20260926 13:39:53.459946 3006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:39:53.460022 3006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:39:53.460129 3006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:53.461390 3006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241452488bed4fb1ad4969113583b95a" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33571 } }
I20260926 13:39:53.461781 3006 leader_election.cc:302] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 241452488bed4fb1ad4969113583b95a; no voters:
I20260926 13:39:53.462054 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:53.446569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.1.49.62:33571)
0926 13:39:53.447043 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:53.447059 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:53.447096 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:39:53.447776 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:53.447795 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:53.448252 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:53.448695 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:53.449460 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:53.450508 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:53.451406 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:53.452016 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:53.452198 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:53.455461 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:53.455466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:53.455482 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:53.455511 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:53.458100 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:53.458764 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:53.458770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:53.458776 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:53.458904 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:53.459284 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:53.459288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:53.459289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:53.459892 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:53.460099 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:53.460547 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 13:39:53.460905 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":151,"threads_started":1}
I20260926 13:39:53.462133 3006 leader_election.cc:288] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:39:53.462281 3011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:39:53.462508 3011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [term 1 LEADER]: Becoming Leader. State: Replica: 241452488bed4fb1ad4969113583b95a, State: Running, Role: LEADER
I20260926 13:39:53.462881 3011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241452488bed4fb1ad4969113583b95a" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33571 } }
I20260926 13:39:53.463418 3006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:39:53.464716 3012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "241452488bed4fb1ad4969113583b95a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241452488bed4fb1ad4969113583b95a" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33571 } } }
I20260926 13:39:53.464850 3012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [sys.catalog]: This master's current role is: LEADER
I20260926 13:39:53.465266 3013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 241452488bed4fb1ad4969113583b95a. Latest consensus state: current_term: 1 leader_uuid: "241452488bed4fb1ad4969113583b95a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241452488bed4fb1ad4969113583b95a" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33571 } } }
I20260926 13:39:53.465376 3013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a [sys.catalog]: This master's current role is: LEADER
W20260926 13:39:53.468736 3026 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:39:53.468880 3026 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:39:53.469020 3021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:39:53.469841 3021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:39:53.476414 3021 catalog_manager.cc:1347] Generated new cluster ID: 348be9ff6f104d1f83dae1d7ddc51554
I20260926 13:39:53.476634 3021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:39:53.517341 3021 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:39:53.518879 3021 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:39:53.523732 3021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a: Generated new TSK 0
I20260926 13:39:53.524569 3021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:39:53.560536 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33571
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:53.684075 3032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:53.684406 3032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:53.684458 3032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:53.688942 3032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:53.689028 3032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:53.689072 3032 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 13:39:53.689088 3032 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:39:53.689159 3032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:39:53.695003 3032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.49.62:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:53.696290 3032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:53.697469 3032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:53.708554 3038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:53.708554 3040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:53.708611 3037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:53.708760 3032 server_base.cc:1034] running on GCE node
I20260926 13:39:53.709259 3032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:53.710062 3032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:53.711267 3032 hybrid_clock.cc:648] HybridClock initialized: now 1790429993711214 us; error 73 us; skew 500 ppm
Sep 26 13:39:53 dist-test-slave-1wns krb5kdc[2915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429993, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:53.714692 3032 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:39:53.715711 3032 webserver.cc:460] Webserver started at http://127.1.49.1:34205/ using document root <none> and password file <none>
I20260926 13:39:53.716408 3032 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:53.716507 3032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:53.716748 3032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:53.719086 3032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "fbaec8f354c14a08991df07d4ed10b56"
format_stamp: "Formatted at 2026-09-26 13:39:53 on dist-test-slave-1wns"
server_key: "58b1d85389754cfcb415f19d9af77706"
server_key_iv: "8090417eeea37ced317aacfd4e81c2b3"
server_key_version: "encryptionkey@0"
I20260926 13:39:53.719730 3032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "fbaec8f354c14a08991df07d4ed10b56"
format_stamp: "Formatted at 2026-09-26 13:39:53 on dist-test-slave-1wns"
server_key: "58b1d85389754cfcb415f19d9af77706"
server_key_iv: "8090417eeea37ced317aacfd4e81c2b3"
server_key_version: "encryptionkey@0"
I20260926 13:39:53.723912 3032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:39:53.726712 3047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:53.728281 3032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:39:53.728436 3032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "fbaec8f354c14a08991df07d4ed10b56"
format_stamp: "Formatted at 2026-09-26 13:39:53 on dist-test-slave-1wns"
server_key: "58b1d85389754cfcb415f19d9af77706"
server_key_iv: "8090417eeea37ced317aacfd4e81c2b3"
server_key_version: "encryptionkey@0"
I20260926 13:39:53.728581 3032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:53.739415 3032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:53.743348 3032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:53.743583 3032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:53.745394 3032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:39:53.745491 3032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:53.745560 3032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:39:53.745608 3032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:39:53 dist-test-slave-1wns krb5kdc[2915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429993, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:53.757519 3032 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:35635
I20260926 13:39:53.757709 3162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:35635 every 8 connection(s)
I20260926 13:39:53.758664 3032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:39:53.762642 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:53.746778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:59073 (local address 127.1.49.62:33571)
0926 13:39:53.746927 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:53.746931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:53.748233 (+ 1302us) server_negotiation.cc:407] Connection header received
0926 13:39:53.749574 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:53.749578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:53.749657 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:53.749737 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:53.751165 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:53.751978 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:53.754161 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:53.754419 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:53.754522 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:53.758329 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:53.758333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:53.758335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:53.758366 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:53.760293 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:53.760887 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:53.760891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:53.760893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:53.760993 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:53.761380 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:53.761383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:53.761385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:53.761614 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:53.761732 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:53.762289 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 13:39:53.762440 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 13:39:53.763671 3056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:53.747342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33571 (local address 127.1.49.1:59073)
0926 13:39:53.748047 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:53.748120 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:53.749315 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:53.749984 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:53.749992 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:53.750393 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:53.750975 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:53.750991 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:53.752159 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:53.752164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:53.753973 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:53.753987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:53.754753 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:53.754757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:53.755028 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:53.756003 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:53.756032 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:53.758069 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:53.760434 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:53.760440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:53.760458 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:53.760760 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:53.761101 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:53.761105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:53.761107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:53.761271 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:53.761739 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:53.761748 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:53.762032 (+ 284us) client_negotiation.cc:769] Sending connection context
0926 13:39:53.762245 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 13:39:53.762644 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":139,"threads_started":1}
I20260926 13:39:53.765122 3163 heartbeater.cc:347] Connected to a master server at 127.1.49.62:33571
I20260926 13:39:53.765473 3163 heartbeater.cc:464] Registering TS with master...
I20260926 13:39:53.766034 3163 heartbeater.cc:511] Master 127.1.49.62:33571 requested a full tablet report, sending...
I20260926 13:39:53.767997 2958 ts_manager.cc:194] Registered new tserver with Master: fbaec8f354c14a08991df07d4ed10b56 (127.1.49.1:35635)
I20260926 13:39:53.768862 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 3032
I20260926 13:39:53.768985 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:39:53.769281 1220 external_mini_cluster.cc:1428] Setting key 729bf279a35f66d69e3fdbb7b0dd5d2c
I20260926 13:39:53.770504 2958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:59073
I20260926 13:39:53.777598 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:53.768306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:49153 (local address 127.1.49.62:33571)
0926 13:39:53.768465 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:53.768472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:53.768693 (+ 221us) server_negotiation.cc:407] Connection header received
0926 13:39:53.768857 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:53.768860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:53.768913 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:53.769017 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:53.769730 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:53.770688 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:53.772031 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:53.772430 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:53.772521 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:53.774225 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:53.774229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:53.774232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:53.774266 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:53.775893 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:53.776348 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:53.776351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:53.776353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:53.776440 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:53.776748 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:53.776751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:53.776752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:53.776933 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:53.777040 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:53.777303 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 13:39:53.777417 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":48}
I20260926 13:39:53.777635 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:53.768284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33571 (local address 127.1.49.1:49153)
0926 13:39:53.768600 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:53.768614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:53.768744 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:53.769160 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:53.769162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:53.769411 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:53.769602 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:53.769612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:53.770855 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:53.770863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:53.771877 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:53.771890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:53.772730 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:53.772736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:53.772887 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:53.773628 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:53.773644 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:53.773998 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:53.776006 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:53.776010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:53.776012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:53.776223 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:53.776531 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:53.776535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:53.776537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:53.776633 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:53.777028 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:53.777031 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:53.777125 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 13:39:53.777238 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 13:39:53.777404 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:39:53.804065 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33571
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:53.927124 3172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:53.927469 3172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:53.927520 3172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:53.931957 3172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:53.932075 3172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:53.932160 3172 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 13:39:53.932185 3172 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:39:53.932220 3172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:39:53.937975 3172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.49.62:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:53.939105 3172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:53.940187 3172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:39:53.951191 3172 server_base.cc:1034] running on GCE node
W20260926 13:39:53.951116 3180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:53.951385 3177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:53.951138 3178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:53.951964 3172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:53.952795 3172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:53.954010 3172 hybrid_clock.cc:648] HybridClock initialized: now 1790429993953993 us; error 32 us; skew 500 ppm
Sep 26 13:39:53 dist-test-slave-1wns krb5kdc[2915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429993, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:53.957373 3172 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:39:53.958458 3172 webserver.cc:460] Webserver started at http://127.1.49.2:45923/ using document root <none> and password file <none>
I20260926 13:39:53.959103 3172 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:53.959182 3172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:53.959419 3172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:53.961809 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "cf1fc9ec2b1448c98557061456f56455"
format_stamp: "Formatted at 2026-09-26 13:39:53 on dist-test-slave-1wns"
server_key: "2615f7fd31acd287cd8c27f8f035536a"
server_key_iv: "2f40ffcf650c9f02b84118c11c597822"
server_key_version: "encryptionkey@0"
I20260926 13:39:53.962437 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "cf1fc9ec2b1448c98557061456f56455"
format_stamp: "Formatted at 2026-09-26 13:39:53 on dist-test-slave-1wns"
server_key: "2615f7fd31acd287cd8c27f8f035536a"
server_key_iv: "2f40ffcf650c9f02b84118c11c597822"
server_key_version: "encryptionkey@0"
I20260926 13:39:53.966952 3172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 13:39:53.969872 3187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:53.971475 3172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:39:53.971645 3172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "cf1fc9ec2b1448c98557061456f56455"
format_stamp: "Formatted at 2026-09-26 13:39:53 on dist-test-slave-1wns"
server_key: "2615f7fd31acd287cd8c27f8f035536a"
server_key_iv: "2f40ffcf650c9f02b84118c11c597822"
server_key_version: "encryptionkey@0"
I20260926 13:39:53.971783 3172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:53.993387 3172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:53.997454 3172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:53.997682 3172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:53.999262 3172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:39:53.999318 3172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:53.999361 3172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:39:53.999420 3172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:54.011281 3172 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:36111
Sep 26 13:39:54 dist-test-slave-1wns krb5kdc[2915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429993, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:54.011355 3302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:36111 every 8 connection(s)
I20260926 13:39:54.012439 3172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:39:54.018523 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.000493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:40219 (local address 127.1.49.62:33571)
0926 13:39:54.000632 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:54.000639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:54.001978 (+ 1339us) server_negotiation.cc:407] Connection header received
0926 13:39:54.003291 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:54.003298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:54.003369 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:54.003480 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:54.005137 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.006017 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.008052 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.008276 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.008371 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.012226 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:54.012231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:54.012233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:54.012264 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:54.014075 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:54.014981 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:54.014986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:54.014988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:54.015091 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:54.016785 (+ 1694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:54.016790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:54.016792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:54.017107 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:54.017226 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:54.018217 (+ 991us) server_negotiation.cc:299] Negotiation successful
0926 13:39:54.018382 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:39:54.019421 3196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.001128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33571 (local address 127.1.49.2:40219)
0926 13:39:54.001784 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:54.001849 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:54.003029 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:54.003699 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:54.003707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:54.004256 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:54.004945 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.004969 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.006176 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.006183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.007905 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.007917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.008459 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.008467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.008854 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.009740 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:54.009763 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:54.011991 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:54.014242 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:54.014247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:54.014260 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:54.014571 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:54.015953 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:54.015956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:54.015958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:54.016531 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:54.017295 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:54.017304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:54.017820 (+ 516us) client_negotiation.cc:769] Sending connection context
0926 13:39:54.018054 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 13:39:54.018417 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":155,"threads_started":1}
I20260926 13:39:54.020751 3303 heartbeater.cc:347] Connected to a master server at 127.1.49.62:33571
I20260926 13:39:54.020908 3303 heartbeater.cc:464] Registering TS with master...
I20260926 13:39:54.021510 3303 heartbeater.cc:511] Master 127.1.49.62:33571 requested a full tablet report, sending...
I20260926 13:39:54.022370 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 3172
I20260926 13:39:54.022501 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:39:54.022774 1220 external_mini_cluster.cc:1428] Setting key 0c3fddd71b86f8ade7a60dd2da1f7940
I20260926 13:39:54.022753 2960 ts_manager.cc:194] Registered new tserver with Master: cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111)
I20260926 13:39:54.023610 2960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:40219
I20260926 13:39:54.032961 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.023560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:51189 (local address 127.1.49.62:33571)
0926 13:39:54.023691 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:54.023696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:54.024343 (+ 647us) server_negotiation.cc:407] Connection header received
0926 13:39:54.024526 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:54.024529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:54.024578 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:54.024669 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:54.025207 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.026032 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.027366 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.027584 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.027671 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.029215 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:54.029218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:54.029221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:54.029244 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:54.030822 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:54.031358 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:54.031361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:54.031363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:54.031442 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:54.031888 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:54.031891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:54.031893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:54.032202 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:54.032351 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:54.032708 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 13:39:54.032826 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:39:54.033088 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.023410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33571 (local address 127.1.49.2:51189)
0926 13:39:54.024257 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:54.024273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:54.024414 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:54.024703 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:54.024707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:54.024933 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:54.025079 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.025087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.026212 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.026217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.027205 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.027233 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.027719 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.027726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.027890 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.028428 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:54.028447 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:54.029012 (+ 565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:54.030954 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:54.030960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:54.030964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:54.031240 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:54.031550 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:54.031555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:54.031558 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:54.031667 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:54.032335 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:54.032342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:54.032476 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 13:39:54.032630 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 13:39:54.032838 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:39:54.056430 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33571
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:39:54.181228 3312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:39:54.181579 3312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:39:54.181675 3312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:39:54.186040 3312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:39:54.186162 3312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:39:54.186244 3312 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 13:39:54.186290 3312 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:39:54.186360 3312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:39:54.192024 3312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.49.62:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:39:54.193240 3312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:39:54.194337 3312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:39:54.204919 3320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:54.204927 3317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:39:54.204926 3318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:39:54.205343 3312 server_base.cc:1034] running on GCE node
I20260926 13:39:54.205902 3312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:39:54.206650 3312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:39:54.207912 3312 hybrid_clock.cc:648] HybridClock initialized: now 1790429994207870 us; error 53 us; skew 500 ppm
Sep 26 13:39:54 dist-test-slave-1wns krb5kdc[2915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429994, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:39:54.211298 3312 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:39:54.212679 3312 webserver.cc:460] Webserver started at http://127.1.49.3:38679/ using document root <none> and password file <none>
I20260926 13:39:54.213376 3312 fs_manager.cc:359] Metadata directory not provided
I20260926 13:39:54.213485 3312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:39:54.213752 3312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:39:54.216238 3312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "2d7528e78bb44af48e8b35fc56e4a24f"
format_stamp: "Formatted at 2026-09-26 13:39:54 on dist-test-slave-1wns"
server_key: "74ae5a38c61d40c20c4e3e523a5c8e25"
server_key_iv: "820a65ee4cd9c135096256a57330cee2"
server_key_version: "encryptionkey@0"
I20260926 13:39:54.216888 3312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "2d7528e78bb44af48e8b35fc56e4a24f"
format_stamp: "Formatted at 2026-09-26 13:39:54 on dist-test-slave-1wns"
server_key: "74ae5a38c61d40c20c4e3e523a5c8e25"
server_key_iv: "820a65ee4cd9c135096256a57330cee2"
server_key_version: "encryptionkey@0"
I20260926 13:39:54.221274 3312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 13:39:54.224108 3327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:54.225546 3312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 13:39:54.225777 3312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "2d7528e78bb44af48e8b35fc56e4a24f"
format_stamp: "Formatted at 2026-09-26 13:39:54 on dist-test-slave-1wns"
server_key: "74ae5a38c61d40c20c4e3e523a5c8e25"
server_key_iv: "820a65ee4cd9c135096256a57330cee2"
server_key_version: "encryptionkey@0"
I20260926 13:39:54.225914 3312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:39:54.237434 3312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:39:54.241197 3312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:39:54.241428 3312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:39:54.243026 3312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:39:54.243105 3312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:39:54.243181 3312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:39:54.243227 3312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:39:54 dist-test-slave-1wns krb5kdc[2915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429994, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:54.255259 3312 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:40027
I20260926 13:39:54.255278 3442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:40027 every 8 connection(s)
I20260926 13:39:54.256351 3312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:39:54.258337 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.244024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:59123 (local address 127.1.49.62:33571)
0926 13:39:54.244156 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:54.244159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:54.245273 (+ 1114us) server_negotiation.cc:407] Connection header received
0926 13:39:54.246405 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:54.246410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:54.246498 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:54.246586 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:54.248201 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.248946 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.250173 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.250489 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.250614 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.253831 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:54.253835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:54.253838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:54.253875 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:54.255846 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:54.256461 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:54.256466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:54.256467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:54.256571 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:54.257054 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:54.257058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:54.257059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:54.257320 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:54.257449 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:54.258002 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 13:39:54.258144 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:39:54.259259 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.244613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33571 (local address 127.1.49.3:59123)
0926 13:39:54.245116 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:54.245161 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:54.246142 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:54.246949 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:54.246956 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:54.247410 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:54.248029 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.248047 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.249083 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.249087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.250026 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.250036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.250682 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.250686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.250895 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.251588 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:54.251619 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:54.253612 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:54.255995 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:54.256004 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:54.256022 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:54.256331 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:54.256709 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:54.256714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:54.256716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:54.256917 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:54.257444 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:54.257451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:54.257732 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 13:39:54.257941 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 13:39:54.258303 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":131,"threads_started":1}
I20260926 13:39:54.260854 3443 heartbeater.cc:347] Connected to a master server at 127.1.49.62:33571
I20260926 13:39:54.261054 3443 heartbeater.cc:464] Registering TS with master...
I20260926 13:39:54.261798 3443 heartbeater.cc:511] Master 127.1.49.62:33571 requested a full tablet report, sending...
I20260926 13:39:54.263190 2958 ts_manager.cc:194] Registered new tserver with Master: 2d7528e78bb44af48e8b35fc56e4a24f (127.1.49.3:40027)
I20260926 13:39:54.264140 2958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:59123
I20260926 13:39:54.264479 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 3312
I20260926 13:39:54.264688 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:39:54.265055 1220 external_mini_cluster.cc:1428] Setting key 5e847012ec376ae8266414781076a40f
I20260926 13:39:54.267339 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:39:54.272360 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.263426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:36999 (local address 127.1.49.62:33571)
0926 13:39:54.263550 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:54.263555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:54.263915 (+ 360us) server_negotiation.cc:407] Connection header received
0926 13:39:54.264068 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:54.264070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:54.264110 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:54.264213 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:54.264827 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.265589 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.266647 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.266984 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.267108 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.268523 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:54.268527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:54.268530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:54.268560 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:54.270443 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:54.270966 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:54.270968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:54.270969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:54.271075 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:54.271384 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:54.271389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:54.271391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:54.271604 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:54.271714 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:54.272056 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 13:39:54.272171 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 13:39:54.272372 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.263296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33571 (local address 127.1.49.3:36999)
0926 13:39:54.263837 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:54.263854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:54.263960 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:54.264254 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:54.264257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:54.264555 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:54.264693 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.264701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.265713 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.265717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.266528 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.266541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.267258 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.267262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.267360 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.267955 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:54.267968 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:54.268360 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:54.270658 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:54.270661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:54.270662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:54.270873 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:54.271185 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:54.271188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:54.271191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:54.271285 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:54.271713 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:54.271716 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:54.271865 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 13:39:54.271984 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 13:39:54.272156 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":122,"threads_started":1}
Sep 26 13:39:54 dist-test-slave-1wns krb5kdc[2915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:39:54 dist-test-slave-1wns krb5kdc[2915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:54.297432 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.287520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.1.49.62:33571)
0926 13:39:54.287659 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:54.287663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:54.287676 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:39:54.287734 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:54.287736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:54.287780 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:54.287914 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:54.288858 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.289392 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.290349 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.290581 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.290673 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.293609 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:54.293613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:54.293616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:54.293662 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:54.295421 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:54.296112 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:54.296116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:54.296118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:54.296208 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:54.296493 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:54.296495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:54.296496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:54.296730 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:54.296849 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:54.297112 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 13:39:54.297220 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:39:54.301033 2958 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:39:54.304523 2958 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:39:54.321679 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.314898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.1.49.3:40027)
0926 13:39:54.316199 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:54.316214 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:54.316240 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:39:54.316323 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:54.316331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:54.316620 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:54.316896 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:54.317170 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.318045 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.319963 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.320871 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.321004 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.321205 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:54.321372 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 13:39:54.321499 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1184,"thread_start_us":134,"threads_started":1}
I20260926 13:39:54.321717 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.314663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:40027 (local address 127.0.0.1:59942)
0926 13:39:54.315335 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:54.315394 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:54.315541 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:54.316797 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:54.316801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:54.316822 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:54.317043 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.317051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.318184 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.318188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.319766 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.319777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.321012 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.321015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.321142 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.321191 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 13:39:54.321324 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 13:39:54.321467 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":72,"thread_start_us":101,"threads_started":1}
I20260926 13:39:54.322428 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.314893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:36111 (local address 127.0.0.1:44302)
0926 13:39:54.315834 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:54.315850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:54.315937 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:54.316686 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:54.316692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:54.316714 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:54.316949 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.316959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.318369 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.318372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.320516 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.320596 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.322005 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.322009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.322106 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.322132 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 13:39:54.322217 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:39:54.322300 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":1342,"thread_start_us":124,"threads_started":1}
I20260926 13:39:54.322716 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.315891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.1.49.1:35635)
0926 13:39:54.316222 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:54.316227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:54.316423 (+ 196us) server_negotiation.cc:407] Connection header received
0926 13:39:54.316681 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:54.316690 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:54.316878 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:54.317015 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:54.317484 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.318519 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.319929 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.320871 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.321010 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.321176 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:54.322188 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0926 13:39:54.322324 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":78,"threads_started":1}
I20260926 13:39:54.322907 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.315659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44302 (local address 127.1.49.2:36111)
0926 13:39:54.316097 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:54.316111 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:54.316137 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:39:54.316235 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:54.316244 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:54.316516 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:54.316684 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:54.317092 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.318246 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.320744 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.321557 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.321769 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.321950 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:54.322670 (+ 720us) server_negotiation.cc:299] Negotiation successful
0926 13:39:54.322773 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":139,"threads_started":1}
I20260926 13:39:54.323261 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.315502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:35635 (local address 127.0.0.1:50154)
0926 13:39:54.316338 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:54.316352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:54.316488 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:54.317194 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:54.317198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:54.317218 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:54.317336 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.317343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.318635 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.318640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.319764 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.319775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.321312 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.321318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.321428 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.321454 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 13:39:54.321553 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 13:39:54.321630 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":51,"thread_start_us":106,"threads_started":1}
I20260926 13:39:54.325313 3377 tablet_service.cc:1467] Processing CreateTablet for tablet f3391da1d8b841e49b55c65446b37940 (DEFAULT_TABLE table=test-table [id=55b1e5e504d241119636a808253dc418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:54.325810 3097 tablet_service.cc:1467] Processing CreateTablet for tablet f3391da1d8b841e49b55c65446b37940 (DEFAULT_TABLE table=test-table [id=55b1e5e504d241119636a808253dc418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:54.325811 3237 tablet_service.cc:1467] Processing CreateTablet for tablet f3391da1d8b841e49b55c65446b37940 (DEFAULT_TABLE table=test-table [id=55b1e5e504d241119636a808253dc418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:54.326472 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3391da1d8b841e49b55c65446b37940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:54.326867 3237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3391da1d8b841e49b55c65446b37940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:54.326867 3097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3391da1d8b841e49b55c65446b37940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:54.334986 3466 tablet_bootstrap.cc:492] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56: Bootstrap starting.
I20260926 13:39:54.335484 3468 tablet_bootstrap.cc:492] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455: Bootstrap starting.
I20260926 13:39:54.335948 3467 tablet_bootstrap.cc:492] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f: Bootstrap starting.
I20260926 13:39:54.338236 3467 tablet_bootstrap.cc:654] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:54.339301 3467 log.cc:824] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:39:54.340705 3468 tablet_bootstrap.cc:654] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:54.341446 3468 log.cc:824] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455: Log is configured to *not* fsync() on all Append() calls
I20260926 13:39:54.341511 3467 tablet_bootstrap.cc:492] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f: No bootstrap required, opened a new log
I20260926 13:39:54.341730 3467 ts_tablet_manager.cc:1397] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:39:54.343858 3468 tablet_bootstrap.cc:492] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455: No bootstrap required, opened a new log
I20260926 13:39:54.343946 3466 tablet_bootstrap.cc:654] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:54.344095 3468 ts_tablet_manager.cc:1397] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 13:39:54.344726 3466 log.cc:824] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56: Log is configured to *not* fsync() on all Append() calls
I20260926 13:39:54.346158 3467 raft_consensus.cc:348] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } }
I20260926 13:39:54.346499 3467 raft_consensus.cc:374] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:54.346565 3467 raft_consensus.cc:729] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7528e78bb44af48e8b35fc56e4a24f, State: Initialized, Role: FOLLOWER
I20260926 13:39:54.346915 3466 tablet_bootstrap.cc:492] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56: No bootstrap required, opened a new log
I20260926 13:39:54.347141 3466 ts_tablet_manager.cc:1397] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260926 13:39:54.347075 3467 consensus_queue.cc:260] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } }
I20260926 13:39:54.348304 3467 ts_tablet_manager.cc:1428] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 13:39:54.348748 3443 heartbeater.cc:503] Master 127.1.49.62:33571 was elected leader, sending a full tablet report...
I20260926 13:39:54.348563 3468 raft_consensus.cc:348] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } }
I20260926 13:39:54.348855 3468 raft_consensus.cc:374] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:54.348922 3468 raft_consensus.cc:729] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1fc9ec2b1448c98557061456f56455, State: Initialized, Role: FOLLOWER
I20260926 13:39:54.349372 3468 consensus_queue.cc:260] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } }
I20260926 13:39:54.350247 3303 heartbeater.cc:503] Master 127.1.49.62:33571 was elected leader, sending a full tablet report...
I20260926 13:39:54.350484 3468 ts_tablet_manager.cc:1428] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:39:54.351423 3466 raft_consensus.cc:348] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } }
I20260926 13:39:54.351756 3466 raft_consensus.cc:374] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:54.351845 3466 raft_consensus.cc:729] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbaec8f354c14a08991df07d4ed10b56, State: Initialized, Role: FOLLOWER
I20260926 13:39:54.352380 3466 consensus_queue.cc:260] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } }
I20260926 13:39:54.353106 3163 heartbeater.cc:503] Master 127.1.49.62:33571 was elected leader, sending a full tablet report...
I20260926 13:39:54.353425 3466 ts_tablet_manager.cc:1428] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 13:39:54.374383 3472 raft_consensus.cc:482] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:39:54.374617 3472 raft_consensus.cc:504] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } }
I20260926 13:39:54.376106 3472 leader_election.cc:288] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111), fbaec8f354c14a08991df07d4ed10b56 (127.1.49.1:35635)
I20260926 13:39:54.381210 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.376336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:36111 (local address 127.1.49.3:43579)
0926 13:39:54.376494 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:54.376512 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:54.376618 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:54.377246 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:54.377250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:54.377271 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:54.377423 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.377434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.378350 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.378354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.379935 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.379948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.380817 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.380820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.380907 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.380929 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:39:54.380992 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:39:54.381063 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 13:39:54.381254 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.376829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:43579 (local address 127.1.49.2:36111)
0926 13:39:54.376974 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:54.376981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:54.376998 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:39:54.377053 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:54.377059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:54.377114 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:54.377199 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:54.377562 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.378232 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.380103 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.380679 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.380816 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.380891 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:54.381024 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 13:39:54.381103 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:39:54.382009 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.376487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:35635 (local address 127.1.49.3:50941)
0926 13:39:54.376903 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:54.376919 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:54.377037 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:54.377376 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:54.377378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:54.377393 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:54.377541 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.377549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.378598 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.378601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.380385 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.380393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.381554 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.381559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.381672 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.381698 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 13:39:54.381805 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 13:39:54.381884 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":147,"threads_started":1}
I20260926 13:39:54.382068 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.376609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:50941 (local address 127.1.49.1:35635)
0926 13:39:54.376754 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:54.376759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:54.377038 (+ 279us) server_negotiation.cc:407] Connection header received
0926 13:39:54.377160 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:54.377193 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:54.377259 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:54.377358 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:54.377689 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.378445 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.380524 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.381260 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.381385 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.381458 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:54.381861 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 13:39:54.381949 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 13:39:54.382155 3257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3391da1d8b841e49b55c65446b37940" candidate_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1fc9ec2b1448c98557061456f56455" is_pre_election: true
I20260926 13:39:54.382511 3257 raft_consensus.cc:2454] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d7528e78bb44af48e8b35fc56e4a24f in term 0.
I20260926 13:39:54.383041 3117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3391da1d8b841e49b55c65446b37940" candidate_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbaec8f354c14a08991df07d4ed10b56" is_pre_election: true
I20260926 13:39:54.383240 3329 leader_election.cc:302] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d7528e78bb44af48e8b35fc56e4a24f, cf1fc9ec2b1448c98557061456f56455; no voters:
I20260926 13:39:54.383422 3117 raft_consensus.cc:2454] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d7528e78bb44af48e8b35fc56e4a24f in term 0.
I20260926 13:39:54.383687 3472 raft_consensus.cc:2781] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:39:54.383837 3472 raft_consensus.cc:482] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:39:54.383905 3472 raft_consensus.cc:3037] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:54.385080 3472 raft_consensus.cc:504] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } }
I20260926 13:39:54.385560 3472 leader_election.cc:288] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [CANDIDATE]: Term 1 election: Requested vote from peers cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111), fbaec8f354c14a08991df07d4ed10b56 (127.1.49.1:35635)
I20260926 13:39:54.386034 3257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3391da1d8b841e49b55c65446b37940" candidate_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1fc9ec2b1448c98557061456f56455"
I20260926 13:39:54.386042 3117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3391da1d8b841e49b55c65446b37940" candidate_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbaec8f354c14a08991df07d4ed10b56"
I20260926 13:39:54.386205 3117 raft_consensus.cc:3037] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:54.386171 3257 raft_consensus.cc:3037] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:54.387497 3257 raft_consensus.cc:2454] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d7528e78bb44af48e8b35fc56e4a24f in term 1.
I20260926 13:39:54.387674 3117 raft_consensus.cc:2454] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d7528e78bb44af48e8b35fc56e4a24f in term 1.
I20260926 13:39:54.387960 3329 leader_election.cc:302] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d7528e78bb44af48e8b35fc56e4a24f, cf1fc9ec2b1448c98557061456f56455; no voters:
I20260926 13:39:54.388193 3472 raft_consensus.cc:2781] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:39:54.388465 3472 raft_consensus.cc:686] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 LEADER]: Becoming Leader. State: Replica: 2d7528e78bb44af48e8b35fc56e4a24f, State: Running, Role: LEADER
I20260926 13:39:54.388880 3472 consensus_queue.cc:237] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } }
I20260926 13:39:54.392534 2960 catalog_manager.cc:5495] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d7528e78bb44af48e8b35fc56e4a24f (127.1.49.3). New cstate: current_term: 1 leader_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } health_report { overall_health: UNKNOWN } } }
Sep 26 13:39:54 dist-test-slave-1wns krb5kdc[2915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429993, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:54.442718 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.433794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33571 (local address 127.0.0.1:42508)
0926 13:39:54.434102 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:54.434115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:54.434250 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:54.434503 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:54.434505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:54.434974 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:39:54.435077 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.435083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.435897 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.435903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.436455 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.436463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.436901 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.436904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.436981 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.437619 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:39:54.437631 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:39:54.439260 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:39:54.440990 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:54.441000 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:54.441007 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:54.441245 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:54.441520 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:39:54.441525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:39:54.441527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:39:54.441647 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:39:54.442039 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:39:54.442048 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:39:54.442159 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 13:39:54.442276 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 13:39:54.442506 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":21,"thread_start_us":122,"threads_started":1}
I20260926 13:39:54.442718 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.433827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.1.49.62:33571)
0926 13:39:54.433993 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:54.433998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:54.434229 (+ 231us) server_negotiation.cc:407] Connection header received
0926 13:39:54.434356 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:54.434359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:54.434403 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:54.434486 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:54.435208 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.435732 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.436575 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.436788 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.436895 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.439413 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:54.439417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:54.439422 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:54.439446 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:54.440859 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:54.441353 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:54.441356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:54.441358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:54.441432 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:54.441732 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:54.441737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:54.441739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:54.441922 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:54.442028 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:54.442320 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 13:39:54.442558 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":59}
I20260926 13:39:54.446997 2960 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:42508:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 13:39:54.447755 2960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:39:54.452993 3377 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6708d5417743b0b810dc2618d39bb8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c39fbcc841d94551be753c9c7bead0ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:39:54.452942 3097 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6708d5417743b0b810dc2618d39bb8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c39fbcc841d94551be753c9c7bead0ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:39:54.453200 3237 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6708d5417743b0b810dc2618d39bb8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c39fbcc841d94551be753c9c7bead0ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:39:54.453778 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6708d5417743b0b810dc2618d39bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:54.453778 3097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6708d5417743b0b810dc2618d39bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:54.453944 3237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6708d5417743b0b810dc2618d39bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:54.456705 3468 tablet_bootstrap.cc:492] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455: Bootstrap starting.
I20260926 13:39:54.456761 3466 tablet_bootstrap.cc:492] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56: Bootstrap starting.
I20260926 13:39:54.456763 3467 tablet_bootstrap.cc:492] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f: Bootstrap starting.
I20260926 13:39:54.457623 3468 tablet_bootstrap.cc:654] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:54.457753 3466 tablet_bootstrap.cc:654] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:54.457753 3467 tablet_bootstrap.cc:654] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:54.458778 3468 tablet_bootstrap.cc:492] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455: No bootstrap required, opened a new log
I20260926 13:39:54.458895 3468 ts_tablet_manager.cc:1397] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:39:54.459194 3466 tablet_bootstrap.cc:492] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56: No bootstrap required, opened a new log
I20260926 13:39:54.459304 3466 ts_tablet_manager.cc:1397] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:39:54.459416 3468 raft_consensus.cc:348] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.459503 3467 tablet_bootstrap.cc:492] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f: No bootstrap required, opened a new log
I20260926 13:39:54.459542 3468 raft_consensus.cc:374] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:54.459578 3467 ts_tablet_manager.cc:1397] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:39:54.459595 3468 raft_consensus.cc:729] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1fc9ec2b1448c98557061456f56455, State: Initialized, Role: FOLLOWER
I20260926 13:39:54.459743 3468 consensus_queue.cc:260] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.459758 3466 raft_consensus.cc:348] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.459939 3466 raft_consensus.cc:374] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:54.459980 3466 raft_consensus.cc:729] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbaec8f354c14a08991df07d4ed10b56, State: Initialized, Role: FOLLOWER
I20260926 13:39:54.460057 3467 raft_consensus.cc:348] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.460111 3466 consensus_queue.cc:260] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.460184 3467 raft_consensus.cc:374] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:54.460067 3473 tablet_replica.cc:439] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } }
I20260926 13:39:54.460215 3467 raft_consensus.cc:729] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7528e78bb44af48e8b35fc56e4a24f, State: Initialized, Role: FOLLOWER
I20260926 13:39:54.460263 3473 tablet_replica.cc:442] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:39:54.460057 3468 ts_tablet_manager.cc:1428] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:39:54.460338 3467 consensus_queue.cc:260] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.460369 3474 tablet_replica.cc:439] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } }
I20260926 13:39:54.460497 3474 tablet_replica.cc:442] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:39:54.460587 3467 ts_tablet_manager.cc:1428] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:39:54.460572 3472 tablet_replica.cc:439] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } }
I20260926 13:39:54.460695 3472 tablet_replica.cc:442] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:39:54.461246 3466 ts_tablet_manager.cc:1428] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260926 13:39:54.507548 3444 tablet.cc:2369] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:39:54.510253 3164 tablet.cc:2369] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:39:54.513687 3304 tablet.cc:2369] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:39:54.673481 3472 raft_consensus.cc:482] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:39:54.673633 3472 raft_consensus.cc:504] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.674088 3472 leader_election.cc:288] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbaec8f354c14a08991df07d4ed10b56 (127.1.49.1:35635), cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111)
I20260926 13:39:54.674723 3257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6708d5417743b0b810dc2618d39bb8" candidate_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1fc9ec2b1448c98557061456f56455" is_pre_election: true
I20260926 13:39:54.674762 3117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6708d5417743b0b810dc2618d39bb8" candidate_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbaec8f354c14a08991df07d4ed10b56" is_pre_election: true
I20260926 13:39:54.674870 3257 raft_consensus.cc:2454] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d7528e78bb44af48e8b35fc56e4a24f in term 0.
I20260926 13:39:54.674981 3117 raft_consensus.cc:2454] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d7528e78bb44af48e8b35fc56e4a24f in term 0.
I20260926 13:39:54.675237 3329 leader_election.cc:302] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d7528e78bb44af48e8b35fc56e4a24f, cf1fc9ec2b1448c98557061456f56455; no voters:
I20260926 13:39:54.675415 3472 raft_consensus.cc:2781] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:39:54.675482 3472 raft_consensus.cc:482] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:39:54.675522 3472 raft_consensus.cc:3037] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:54.676694 3472 raft_consensus.cc:504] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.677132 3472 leader_election.cc:288] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [CANDIDATE]: Term 1 election: Requested vote from peers fbaec8f354c14a08991df07d4ed10b56 (127.1.49.1:35635), cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111)
I20260926 13:39:54.677560 3257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6708d5417743b0b810dc2618d39bb8" candidate_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1fc9ec2b1448c98557061456f56455"
I20260926 13:39:54.677677 3257 raft_consensus.cc:3037] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:54.677734 3117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6708d5417743b0b810dc2618d39bb8" candidate_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbaec8f354c14a08991df07d4ed10b56"
I20260926 13:39:54.677917 3117 raft_consensus.cc:3037] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:54.678735 3257 raft_consensus.cc:2454] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d7528e78bb44af48e8b35fc56e4a24f in term 1.
I20260926 13:39:54.679036 3117 raft_consensus.cc:2454] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d7528e78bb44af48e8b35fc56e4a24f in term 1.
I20260926 13:39:54.679128 3329 leader_election.cc:302] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d7528e78bb44af48e8b35fc56e4a24f, cf1fc9ec2b1448c98557061456f56455; no voters:
I20260926 13:39:54.679327 3472 raft_consensus.cc:2781] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:39:54.679440 3472 raft_consensus.cc:686] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 LEADER]: Becoming Leader. State: Replica: 2d7528e78bb44af48e8b35fc56e4a24f, State: Running, Role: LEADER
I20260926 13:39:54.679594 3472 consensus_queue.cc:237] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.680800 3476 tablet_replica.cc:439] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f: TxnStatusTablet state changed. Reason: New leader 2d7528e78bb44af48e8b35fc56e4a24f. Latest consensus state: current_term: 1 leader_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } }
I20260926 13:39:54.680939 3476 tablet_replica.cc:442] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:39:54.681268 3488 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 13:39:54.681528 3488 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 13:39:54.681749 2959 catalog_manager.cc:5495] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d7528e78bb44af48e8b35fc56e4a24f (127.1.49.3). New cstate: current_term: 1 leader_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } health_report { overall_health: HEALTHY } } }
I20260926 13:39:54.705736 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.700490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.1.49.3:40027)
0926 13:39:54.700757 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:54.700763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:54.700787 (+ 24us) server_negotiation.cc:407] Connection header received
0926 13:39:54.700974 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:54.700978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:54.701059 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:54.701189 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:39:54.701448 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.702450 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.703622 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.704021 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.704171 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.704598 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:39:54.705267 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:39:54.705388 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:54.705513 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 13:39:54.705581 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 13:39:54.705766 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.700346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:40027 (local address 127.0.0.1:59950)
0926 13:39:54.700596 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:54.700677 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:54.700810 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:54.701217 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:54.701220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:54.701227 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:39:54.701331 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.701336 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.702644 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.702647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.703333 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.703407 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.704232 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.704239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.704358 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.704426 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:39:54.705383 (+ 957us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:39:54.705406 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:39:54.705491 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:39:54.705606 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 13:39:54.711702 3257 raft_consensus.cc:1261] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Refusing update from remote peer 2d7528e78bb44af48e8b35fc56e4a24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:54.711776 3117 raft_consensus.cc:1261] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56 [term 1 FOLLOWER]: Refusing update from remote peer 2d7528e78bb44af48e8b35fc56e4a24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:54.712543 3472 consensus_queue.cc:1035] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:54.712815 3476 consensus_queue.cc:1035] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:54.717293 3474 tablet_replica.cc:439] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56: TxnStatusTablet state changed. Reason: New leader 2d7528e78bb44af48e8b35fc56e4a24f. Latest consensus state: current_term: 1 leader_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } }
I20260926 13:39:54.717504 3474 tablet_replica.cc:442] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:39:54.717717 3473 tablet_replica.cc:439] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455: TxnStatusTablet state changed. Reason: New leader 2d7528e78bb44af48e8b35fc56e4a24f. Latest consensus state: current_term: 1 leader_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } }
I20260926 13:39:54.717849 3473 tablet_replica.cc:442] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:39:54.721985 3476 tablet_replica.cc:439] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } }
I20260926 13:39:54.722146 3476 tablet_replica.cc:442] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:39:54.722378 3473 tablet_replica.cc:439] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } }
I20260926 13:39:54.722517 3473 tablet_replica.cc:442] T 4b6708d5417743b0b810dc2618d39bb8 P cf1fc9ec2b1448c98557061456f56455: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:39:54.723172 3492 mvcc.cc:205] Tried to move back new op lower bound from 7333601258331230208 to 7333601258209361920. Current Snapshot: MvccSnapshot[applied={T|T < 7333601258331230208}]
I20260926 13:39:54.723510 3472 tablet_replica.cc:439] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } }
I20260926 13:39:54.723623 3472 tablet_replica.cc:442] T 4b6708d5417743b0b810dc2618d39bb8 P 2d7528e78bb44af48e8b35fc56e4a24f: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:39:54.725080 3474 tablet_replica.cc:439] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } } }
I20260926 13:39:54.725208 3474 tablet_replica.cc:442] T 4b6708d5417743b0b810dc2618d39bb8 P fbaec8f354c14a08991df07d4ed10b56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:39:54.729224 3493 mvcc.cc:205] Tried to move back new op lower bound from 7333601258331230208 to 7333601258209361920. Current Snapshot: MvccSnapshot[applied={T|T < 7333601258331230208}]
I20260926 13:39:54.735538 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.731535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.1.49.3:40027)
0926 13:39:54.731652 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:54.731656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:54.731693 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:39:54.731847 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:54.731850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:54.731903 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:54.731987 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:39:54.732283 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.733226 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.734272 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.734614 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.734743 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.735038 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:39:54.735131 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:39:54.735202 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:54.735325 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 13:39:54.735380 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:39:54.748883 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.744858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:40027 (local address 127.1.49.3:60805)
0926 13:39:54.745093 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:54.745106 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:54.745204 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:54.745449 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:54.745451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:54.745456 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:39:54.745568 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.745572 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.746585 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.746588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.747257 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:54.747265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.747993 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.747997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:54.748114 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.748170 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:39:54.748552 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:39:54.748558 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:39:54.748635 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 13:39:54.748718 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 13:39:54.748886 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:54.744991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:60805 (local address 127.1.49.3:40027)
0926 13:39:54.745100 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:54.745104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:54.745177 (+ 73us) server_negotiation.cc:407] Connection header received
0926 13:39:54.745300 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:54.745303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:54.745354 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:54.745434 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:39:54.745666 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.746447 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.747380 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:54.747850 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:54.747961 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:54.748345 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:39:54.748436 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:39:54.748509 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:54.748668 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 13:39:54.748725 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:39:54.757705 3257 raft_consensus.cc:1261] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Refusing update from remote peer 2d7528e78bb44af48e8b35fc56e4a24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:54.757755 3117 raft_consensus.cc:1261] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56 [term 1 FOLLOWER]: Refusing update from remote peer 2d7528e78bb44af48e8b35fc56e4a24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:54.758267 3472 consensus_queue.cc:1035] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:54.758522 3476 consensus_queue.cc:1035] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:54.765466 3491 mvcc.cc:205] Tried to move back new op lower bound from 7333601258523217920 to 7333601257020039168. Current Snapshot: MvccSnapshot[applied={T|T < 7333601258523217920}]
I20260926 13:39:54.814802 2959 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:39:54.815013 2959 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:39:54.817948 2959 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a: Sending DeleteTablet for 3 replicas of tablet f3391da1d8b841e49b55c65446b37940
I20260926 13:39:54.819067 3237 tablet_service.cc:1514] Processing DeleteTablet for tablet f3391da1d8b841e49b55c65446b37940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:54 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:44302
I20260926 13:39:54.819185 3097 tablet_service.cc:1514] Processing DeleteTablet for tablet f3391da1d8b841e49b55c65446b37940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:54 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:50154
I20260926 13:39:54.819720 3510 tablet_replica.cc:330] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455: stopping tablet replica
I20260926 13:39:54.819971 3511 tablet_replica.cc:330] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56: stopping tablet replica
I20260926 13:39:54.819994 3377 tablet_service.cc:1514] Processing DeleteTablet for tablet f3391da1d8b841e49b55c65446b37940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:54 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:59942
I20260926 13:39:54.820129 3510 raft_consensus.cc:2229] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:39:54.820376 3510 raft_consensus.cc:2258] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:54.820528 3511 raft_consensus.cc:2229] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:39:54.820534 2959 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 13:39:54.820938 3511 raft_consensus.cc:2258] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 13:39:54.821043 2959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:39:54.822160 3512 tablet_replica.cc:330] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f: stopping tablet replica
I20260926 13:39:54.822149 3510 ts_tablet_manager.cc:1905] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:54.822427 3512 raft_consensus.cc:2229] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:39:54.822820 3512 raft_consensus.cc:2258] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:54.824391 3512 ts_tablet_manager.cc:1905] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:54.825511 3510 ts_tablet_manager.cc:1918] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:39:54.825613 3510 log.cc:1196] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/f3391da1d8b841e49b55c65446b37940
I20260926 13:39:54.825949 3510 ts_tablet_manager.cc:1939] T f3391da1d8b841e49b55c65446b37940 P cf1fc9ec2b1448c98557061456f56455: Deleting consensus metadata
I20260926 13:39:54.827565 2945 catalog_manager.cc:4841] TS cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111): tablet f3391da1d8b841e49b55c65446b37940 (table test-table [id=55b1e5e504d241119636a808253dc418]) successfully deleted
I20260926 13:39:54.827900 3237 tablet_service.cc:1467] Processing CreateTablet for tablet 6f907966c7f54dd0af40babe3a316926 (DEFAULT_TABLE table=test-table [id=38f32db746f5445a8ff535034cfec727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:54.828169 3237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f907966c7f54dd0af40babe3a316926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:54.828328 3377 tablet_service.cc:1467] Processing CreateTablet for tablet 6f907966c7f54dd0af40babe3a316926 (DEFAULT_TABLE table=test-table [id=38f32db746f5445a8ff535034cfec727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:54.828357 3097 tablet_service.cc:1467] Processing CreateTablet for tablet 6f907966c7f54dd0af40babe3a316926 (DEFAULT_TABLE table=test-table [id=38f32db746f5445a8ff535034cfec727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:54.828588 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f907966c7f54dd0af40babe3a316926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:54.828640 3097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f907966c7f54dd0af40babe3a316926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:54.831130 3468 tablet_bootstrap.cc:492] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455: Bootstrap starting.
I20260926 13:39:54.832000 3466 tablet_bootstrap.cc:492] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56: Bootstrap starting.
I20260926 13:39:54.832267 3468 tablet_bootstrap.cc:654] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:54.832852 3466 tablet_bootstrap.cc:654] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:54.832998 3467 tablet_bootstrap.cc:492] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f: Bootstrap starting.
I20260926 13:39:54.833379 3511 ts_tablet_manager.cc:1905] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:54.834570 3467 tablet_bootstrap.cc:654] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:54.835899 3467 tablet_bootstrap.cc:492] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f: No bootstrap required, opened a new log
I20260926 13:39:54.836001 3467 ts_tablet_manager.cc:1397] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:39:54.836526 3467 raft_consensus.cc:348] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.836692 3467 raft_consensus.cc:374] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:54.836743 3467 raft_consensus.cc:729] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7528e78bb44af48e8b35fc56e4a24f, State: Initialized, Role: FOLLOWER
I20260926 13:39:54.836905 3467 consensus_queue.cc:260] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.837211 3467 ts_tablet_manager.cc:1428] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:39:54.838374 3512 ts_tablet_manager.cc:1918] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:39:54.838493 3512 log.cc:1196] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/f3391da1d8b841e49b55c65446b37940
I20260926 13:39:54.839430 3512 ts_tablet_manager.cc:1939] T f3391da1d8b841e49b55c65446b37940 P 2d7528e78bb44af48e8b35fc56e4a24f: Deleting consensus metadata
I20260926 13:39:54.836987 3511 ts_tablet_manager.cc:1918] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:39:54.839869 3511 log.cc:1196] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/wal/wals/f3391da1d8b841e49b55c65446b37940
I20260926 13:39:54.840440 3511 ts_tablet_manager.cc:1939] T f3391da1d8b841e49b55c65446b37940 P fbaec8f354c14a08991df07d4ed10b56: Deleting consensus metadata
I20260926 13:39:54.840687 3468 tablet_bootstrap.cc:492] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455: No bootstrap required, opened a new log
I20260926 13:39:54.840792 3468 ts_tablet_manager.cc:1397] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260926 13:39:54.840863 2944 catalog_manager.cc:4841] TS 2d7528e78bb44af48e8b35fc56e4a24f (127.1.49.3:40027): tablet f3391da1d8b841e49b55c65446b37940 (table test-table [id=55b1e5e504d241119636a808253dc418]) successfully deleted
I20260926 13:39:54.841267 3468 raft_consensus.cc:348] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.841405 3468 raft_consensus.cc:374] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:54.841439 3468 raft_consensus.cc:729] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1fc9ec2b1448c98557061456f56455, State: Initialized, Role: FOLLOWER
I20260926 13:39:54.841590 3468 consensus_queue.cc:260] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.841902 3468 ts_tablet_manager.cc:1428] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 13:39:54.841898 2944 catalog_manager.cc:4841] TS fbaec8f354c14a08991df07d4ed10b56 (127.1.49.1:35635): tablet f3391da1d8b841e49b55c65446b37940 (table test-table [id=55b1e5e504d241119636a808253dc418]) successfully deleted
I20260926 13:39:54.842963 3466 tablet_bootstrap.cc:492] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56: No bootstrap required, opened a new log
I20260926 13:39:54.843077 3466 ts_tablet_manager.cc:1397] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260926 13:39:54.843587 3466 raft_consensus.cc:348] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.843720 3466 raft_consensus.cc:374] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:54.843746 3466 raft_consensus.cc:729] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbaec8f354c14a08991df07d4ed10b56, State: Initialized, Role: FOLLOWER
I20260926 13:39:54.843919 3466 consensus_queue.cc:260] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:54.844197 3466 ts_tablet_manager.cc:1428] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 13:39:55.004163 3474 raft_consensus.cc:482] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:39:55.004443 3474 raft_consensus.cc:504] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:55.005789 3474 leader_election.cc:288] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111), 2d7528e78bb44af48e8b35fc56e4a24f (127.1.49.3:40027)
I20260926 13:39:55.010660 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:55.006202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:54439 (local address 127.1.49.3:40027)
0926 13:39:55.006347 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:55.006352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:55.006488 (+ 136us) server_negotiation.cc:407] Connection header received
0926 13:39:55.006736 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:55.006738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:55.006778 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:55.006819 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:55.007346 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.008171 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.009600 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.010050 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.010180 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:55.010250 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:55.010435 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 13:39:55.010516 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 13:39:55.010679 3516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:55.006054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:40027 (local address 127.1.49.1:54439)
0926 13:39:55.006376 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:55.006395 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:55.006597 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:55.006959 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:55.006963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:55.006991 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:55.007205 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:55.007227 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.008307 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.008311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:55.009441 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:55.009453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.010189 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.010192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:55.010297 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:55.010323 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 13:39:55.010417 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 13:39:55.010506 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":61,"spinlock_wait_cycles":11136,"thread_start_us":78,"threads_started":1}
I20260926 13:39:55.011410 3397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f907966c7f54dd0af40babe3a316926" candidate_uuid: "fbaec8f354c14a08991df07d4ed10b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" is_pre_election: true
I20260926 13:39:55.011744 3397 raft_consensus.cc:2454] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbaec8f354c14a08991df07d4ed10b56 in term 0.
I20260926 13:39:55.011984 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:55.006054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:36111 (local address 127.1.49.1:49529)
0926 13:39:55.006474 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:55.006488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:55.006596 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:55.006959 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:55.006963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:55.006993 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:39:55.007188 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:55.007198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.008890 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.008894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:55.010138 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:55.010149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.011486 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.011490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:55.011583 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:55.011620 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 13:39:55.011700 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 13:39:55.011782 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":38,"spinlock_wait_cycles":35456,"thread_start_us":83,"threads_started":1}
I20260926 13:39:55.012001 3518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:55.006446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:49529 (local address 127.1.49.2:36111)
0926 13:39:55.006697 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:55.006709 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:55.006725 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:39:55.006779 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:55.006781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:55.006832 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:55.006932 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:39:55.007970 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.008726 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.010594 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.011229 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.011336 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:55.011400 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:55.011744 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 13:39:55.011869 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260926 13:39:55.012471 3257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f907966c7f54dd0af40babe3a316926" candidate_uuid: "fbaec8f354c14a08991df07d4ed10b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1fc9ec2b1448c98557061456f56455" is_pre_election: true
I20260926 13:39:55.012627 3257 raft_consensus.cc:2454] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbaec8f354c14a08991df07d4ed10b56 in term 0.
I20260926 13:39:55.012751 3048 leader_election.cc:302] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d7528e78bb44af48e8b35fc56e4a24f, fbaec8f354c14a08991df07d4ed10b56; no voters:
I20260926 13:39:55.013207 3474 raft_consensus.cc:2781] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:39:55.013255 3474 raft_consensus.cc:482] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:39:55.013281 3474 raft_consensus.cc:3037] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:55.014338 3474 raft_consensus.cc:504] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:55.014729 3474 leader_election.cc:288] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [CANDIDATE]: Term 1 election: Requested vote from peers cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111), 2d7528e78bb44af48e8b35fc56e4a24f (127.1.49.3:40027)
I20260926 13:39:55.015065 3257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f907966c7f54dd0af40babe3a316926" candidate_uuid: "fbaec8f354c14a08991df07d4ed10b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1fc9ec2b1448c98557061456f56455"
I20260926 13:39:55.015224 3257 raft_consensus.cc:3037] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:55.015580 3397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f907966c7f54dd0af40babe3a316926" candidate_uuid: "fbaec8f354c14a08991df07d4ed10b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7528e78bb44af48e8b35fc56e4a24f"
I20260926 13:39:55.015703 3397 raft_consensus.cc:3037] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:55.016758 3257 raft_consensus.cc:2454] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbaec8f354c14a08991df07d4ed10b56 in term 1.
I20260926 13:39:55.017316 3397 raft_consensus.cc:2454] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbaec8f354c14a08991df07d4ed10b56 in term 1.
I20260926 13:39:55.017316 3049 leader_election.cc:302] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf1fc9ec2b1448c98557061456f56455, fbaec8f354c14a08991df07d4ed10b56; no voters:
I20260926 13:39:55.017931 3474 raft_consensus.cc:2781] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:39:55.018153 3474 raft_consensus.cc:686] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [term 1 LEADER]: Becoming Leader. State: Replica: fbaec8f354c14a08991df07d4ed10b56, State: Running, Role: LEADER
I20260926 13:39:55.018443 3474 consensus_queue.cc:237] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:55.021374 2959 catalog_manager.cc:5495] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbaec8f354c14a08991df07d4ed10b56 (127.1.49.1). New cstate: current_term: 1 leader_uuid: "fbaec8f354c14a08991df07d4ed10b56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } health_report { overall_health: UNKNOWN } } }
I20260926 13:39:55.042121 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:55.037680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.1.49.1:35635)
0926 13:39:55.038007 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:55.038014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:55.038026 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:39:55.038101 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:55.038103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:55.038158 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:55.038301 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:39:55.038547 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.039112 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.040113 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.040713 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.040846 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:55.041148 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:39:55.041662 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:39:55.041805 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:55.041868 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 13:39:55.041935 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":115,"threads_started":1}
I20260926 13:39:55.056180 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:55.051868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:59193 (local address 127.1.49.3:40027)
0926 13:39:55.052005 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:55.052008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:55.052025 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:39:55.052080 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:55.052082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:55.052126 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:55.052201 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:39:55.052843 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.053393 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.054633 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.054903 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.055085 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:55.055403 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:39:55.055592 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:39:55.055680 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:55.055990 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 13:39:55.056060 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 13:39:55.056180 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:55.051281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:40027 (local address 127.1.49.1:59193)
0926 13:39:55.051677 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:55.051697 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:55.051897 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:55.052440 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:55.052443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:55.052452 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:39:55.052674 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:55.052680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.053614 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.053618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:55.054380 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:55.054392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.055060 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.055064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:55.055158 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:55.055227 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:39:55.055741 (+ 514us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:39:55.055749 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 13:39:55.055919 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 13:39:55.056005 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":172,"threads_started":1}
I20260926 13:39:55.064038 3257 raft_consensus.cc:1261] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Refusing update from remote peer fbaec8f354c14a08991df07d4ed10b56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:55.064167 3397 raft_consensus.cc:1261] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Refusing update from remote peer fbaec8f354c14a08991df07d4ed10b56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:55.064636 3519 consensus_queue.cc:1035] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:55.064858 3474 consensus_queue.cc:1035] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:55.084118 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:55.080949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:45643 (local address 127.1.49.1:35635)
0926 13:39:55.081092 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:55.081098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:55.081113 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:39:55.081155 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:55.081158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:55.081247 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:55.081328 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:39:55.081607 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.082165 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.083015 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.083241 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.083321 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:55.083602 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:39:55.083675 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:39:55.083733 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:55.083897 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 13:39:55.083959 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:39:55.084137 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:55.080780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:35635 (local address 127.1.49.3:45643)
0926 13:39:55.080907 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:55.080922 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:39:55.081030 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:39:55.081358 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:39:55.081361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:39:55.081369 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:39:55.081477 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:55.081484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.082267 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.082270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:55.082880 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:39:55.082889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:39:55.083335 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:39:55.083341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:39:55.083433 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:55.083462 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:39:55.083762 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:39:55.083769 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:39:55.083879 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 13:39:55.083980 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 13:39:55.085038 3507 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 13:39:56.651604 2958 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:39:56.651782 2958 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:39:56.654728 2958 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a: Sending DeleteTablet for 3 replicas of tablet 6f907966c7f54dd0af40babe3a316926
I20260926 13:39:56.655479 3097 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f907966c7f54dd0af40babe3a316926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:56 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:50154
I20260926 13:39:56.655540 3237 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f907966c7f54dd0af40babe3a316926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:56 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:44302
I20260926 13:39:56.655534 3377 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f907966c7f54dd0af40babe3a316926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:56 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:59942
I20260926 13:39:56.655948 3551 tablet_replica.cc:330] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f: stopping tablet replica
I20260926 13:39:56.655965 3550 tablet_replica.cc:330] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56: stopping tablet replica
I20260926 13:39:56.656098 3550 raft_consensus.cc:2229] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:39:56.656147 3551 raft_consensus.cc:2229] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:39:56.656368 3551 raft_consensus.cc:2258] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:56.656472 3550 raft_consensus.cc:2258] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:56.656736 3552 tablet_replica.cc:330] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455: stopping tablet replica
I20260926 13:39:56.656895 3552 raft_consensus.cc:2229] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:39:56.657049 3552 raft_consensus.cc:2258] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:56.657389 3550 ts_tablet_manager.cc:1905] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:56.657701 3552 ts_tablet_manager.cc:1905] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:56.660421 3552 ts_tablet_manager.cc:1918] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:39:56.660458 3550 ts_tablet_manager.cc:1918] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:39:56.660557 3552 log.cc:1196] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/6f907966c7f54dd0af40babe3a316926
I20260926 13:39:56.660585 3551 ts_tablet_manager.cc:1905] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:56.660655 3550 log.cc:1196] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/wal/wals/6f907966c7f54dd0af40babe3a316926
I20260926 13:39:56.660881 3552 ts_tablet_manager.cc:1939] T 6f907966c7f54dd0af40babe3a316926 P cf1fc9ec2b1448c98557061456f56455: Deleting consensus metadata
I20260926 13:39:56.661144 3550 ts_tablet_manager.cc:1939] T 6f907966c7f54dd0af40babe3a316926 P fbaec8f354c14a08991df07d4ed10b56: Deleting consensus metadata
I20260926 13:39:56.661772 2945 catalog_manager.cc:4841] TS cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111): tablet 6f907966c7f54dd0af40babe3a316926 (table test-table [id=38f32db746f5445a8ff535034cfec727]) successfully deleted
I20260926 13:39:56.662094 2944 catalog_manager.cc:4841] TS fbaec8f354c14a08991df07d4ed10b56 (127.1.49.1:35635): tablet 6f907966c7f54dd0af40babe3a316926 (table test-table [id=38f32db746f5445a8ff535034cfec727]) successfully deleted
I20260926 13:39:56.664073 3551 ts_tablet_manager.cc:1918] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:39:56.664151 3551 log.cc:1196] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/6f907966c7f54dd0af40babe3a316926
I20260926 13:39:56.664480 3551 ts_tablet_manager.cc:1939] T 6f907966c7f54dd0af40babe3a316926 P 2d7528e78bb44af48e8b35fc56e4a24f: Deleting consensus metadata
I20260926 13:39:56.665372 2944 catalog_manager.cc:4841] TS 2d7528e78bb44af48e8b35fc56e4a24f (127.1.49.3:40027): tablet 6f907966c7f54dd0af40babe3a316926 (table test-table [id=38f32db746f5445a8ff535034cfec727]) successfully deleted
Sep 26 13:39:56 dist-test-slave-1wns krb5kdc[2915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429996, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:39:56 dist-test-slave-1wns krb5kdc[2915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429996, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:39:56.687604 3560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:56.677812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42516 (local address 127.1.49.62:33571)
0926 13:39:56.678084 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:56.678091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:56.678105 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:39:56.678150 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:56.678152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:56.678194 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:56.678284 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:39:56.679351 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:56.680016 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:56.681044 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:56.681276 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:56.681364 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:56.684171 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:39:56.684175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:39:56.684177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:39:56.684206 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:39:56.685867 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:56.686324 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:56.686327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:56.686328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:56.686407 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:39:56.686681 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:39:56.686683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:39:56.686685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:39:56.686875 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:39:56.686981 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:56.687283 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 13:39:56.687400 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":119,"threads_started":1}
I20260926 13:39:56.690817 2958 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:39:56.691326 2958 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:39:56.696698 3237 tablet_service.cc:1467] Processing CreateTablet for tablet 1d18fa3f641f4bfb86f6b1961146f2a9 (DEFAULT_TABLE table=test-table [id=aa1603abd418428db35efc674bc578f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:56.696851 3377 tablet_service.cc:1467] Processing CreateTablet for tablet 1d18fa3f641f4bfb86f6b1961146f2a9 (DEFAULT_TABLE table=test-table [id=aa1603abd418428db35efc674bc578f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:56.697045 3237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d18fa3f641f4bfb86f6b1961146f2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:56.697080 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d18fa3f641f4bfb86f6b1961146f2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:56.700057 3097 tablet_service.cc:1467] Processing CreateTablet for tablet 1d18fa3f641f4bfb86f6b1961146f2a9 (DEFAULT_TABLE table=test-table [id=aa1603abd418428db35efc674bc578f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:56.700402 3097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d18fa3f641f4bfb86f6b1961146f2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:56.700510 3562 tablet_bootstrap.cc:492] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455: Bootstrap starting.
I20260926 13:39:56.701334 3562 tablet_bootstrap.cc:654] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:56.702051 3561 tablet_bootstrap.cc:492] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f: Bootstrap starting.
I20260926 13:39:56.702539 3562 tablet_bootstrap.cc:492] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455: No bootstrap required, opened a new log
I20260926 13:39:56.702636 3562 ts_tablet_manager.cc:1397] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:39:56.702656 3564 tablet_bootstrap.cc:492] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56: Bootstrap starting.
I20260926 13:39:56.703393 3562 raft_consensus.cc:348] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:56.703538 3562 raft_consensus.cc:374] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:56.703584 3562 raft_consensus.cc:729] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1fc9ec2b1448c98557061456f56455, State: Initialized, Role: FOLLOWER
I20260926 13:39:56.703655 3564 tablet_bootstrap.cc:654] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:56.703853 3561 tablet_bootstrap.cc:654] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:56.703765 3562 consensus_queue.cc:260] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:56.704387 3562 ts_tablet_manager.cc:1428] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:39:56.705276 3561 tablet_bootstrap.cc:492] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f: No bootstrap required, opened a new log
I20260926 13:39:56.705374 3561 ts_tablet_manager.cc:1397] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:39:56.705751 3564 tablet_bootstrap.cc:492] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56: No bootstrap required, opened a new log
I20260926 13:39:56.705745 3561 raft_consensus.cc:348] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:56.705858 3561 raft_consensus.cc:374] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:56.705873 3564 ts_tablet_manager.cc:1397] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 13:39:56.705899 3561 raft_consensus.cc:729] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7528e78bb44af48e8b35fc56e4a24f, State: Initialized, Role: FOLLOWER
I20260926 13:39:56.706048 3561 consensus_queue.cc:260] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:56.706413 3564 raft_consensus.cc:348] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:56.706542 3564 raft_consensus.cc:374] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:56.706591 3564 raft_consensus.cc:729] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbaec8f354c14a08991df07d4ed10b56, State: Initialized, Role: FOLLOWER
I20260926 13:39:56.706743 3564 consensus_queue.cc:260] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:56.706835 3561 ts_tablet_manager.cc:1428] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:39:56.707098 3564 ts_tablet_manager.cc:1428] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:39:56.890718 3544 raft_consensus.cc:482] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:39:56.890996 3544 raft_consensus.cc:504] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:56.891554 3544 leader_election.cc:288] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111), 2d7528e78bb44af48e8b35fc56e4a24f (127.1.49.3:40027)
I20260926 13:39:56.892138 3257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d18fa3f641f4bfb86f6b1961146f2a9" candidate_uuid: "fbaec8f354c14a08991df07d4ed10b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1fc9ec2b1448c98557061456f56455" is_pre_election: true
I20260926 13:39:56.892200 3397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d18fa3f641f4bfb86f6b1961146f2a9" candidate_uuid: "fbaec8f354c14a08991df07d4ed10b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" is_pre_election: true
I20260926 13:39:56.892330 3397 raft_consensus.cc:2454] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbaec8f354c14a08991df07d4ed10b56 in term 0.
I20260926 13:39:56.892326 3257 raft_consensus.cc:2454] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbaec8f354c14a08991df07d4ed10b56 in term 0.
I20260926 13:39:56.892732 3048 leader_election.cc:302] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d7528e78bb44af48e8b35fc56e4a24f, cf1fc9ec2b1448c98557061456f56455, fbaec8f354c14a08991df07d4ed10b56; no voters:
I20260926 13:39:56.892911 3544 raft_consensus.cc:2781] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:39:56.892993 3544 raft_consensus.cc:482] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:39:56.893038 3544 raft_consensus.cc:3037] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:56.894085 3544 raft_consensus.cc:504] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:56.894428 3544 leader_election.cc:288] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [CANDIDATE]: Term 1 election: Requested vote from peers cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111), 2d7528e78bb44af48e8b35fc56e4a24f (127.1.49.3:40027)
I20260926 13:39:56.894814 3257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d18fa3f641f4bfb86f6b1961146f2a9" candidate_uuid: "fbaec8f354c14a08991df07d4ed10b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1fc9ec2b1448c98557061456f56455"
I20260926 13:39:56.894840 3397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d18fa3f641f4bfb86f6b1961146f2a9" candidate_uuid: "fbaec8f354c14a08991df07d4ed10b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7528e78bb44af48e8b35fc56e4a24f"
I20260926 13:39:56.894950 3397 raft_consensus.cc:3037] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:56.894963 3257 raft_consensus.cc:3037] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:56.896136 3257 raft_consensus.cc:2454] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbaec8f354c14a08991df07d4ed10b56 in term 1.
I20260926 13:39:56.896136 3397 raft_consensus.cc:2454] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbaec8f354c14a08991df07d4ed10b56 in term 1.
I20260926 13:39:56.896468 3049 leader_election.cc:302] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf1fc9ec2b1448c98557061456f56455, fbaec8f354c14a08991df07d4ed10b56; no voters:
I20260926 13:39:56.896711 3544 raft_consensus.cc:2781] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:39:56.896803 3544 raft_consensus.cc:686] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [term 1 LEADER]: Becoming Leader. State: Replica: fbaec8f354c14a08991df07d4ed10b56, State: Running, Role: LEADER
I20260926 13:39:56.896979 3544 consensus_queue.cc:237] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:56.898834 2959 catalog_manager.cc:5495] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbaec8f354c14a08991df07d4ed10b56 (127.1.49.1). New cstate: current_term: 1 leader_uuid: "fbaec8f354c14a08991df07d4ed10b56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } health_report { overall_health: UNKNOWN } } }
I20260926 13:39:56.951464 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:56.947985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.1.49.1:35635)
0926 13:39:56.948333 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:56.948340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:56.948352 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:39:56.948396 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:56.948398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:56.948440 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:56.948538 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:39:56.948886 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:56.949454 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:56.950335 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:56.950582 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:56.950696 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:56.950969 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:39:56.951056 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:39:56.951141 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:56.951227 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 13:39:56.951284 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":145,"threads_started":1}
I20260926 13:39:56.959141 3397 raft_consensus.cc:1261] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Refusing update from remote peer fbaec8f354c14a08991df07d4ed10b56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:56.959214 3257 raft_consensus.cc:1261] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Refusing update from remote peer fbaec8f354c14a08991df07d4ed10b56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:56.959637 3543 consensus_queue.cc:1035] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:56.959837 3544 consensus_queue.cc:1035] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:56.997404 2960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:39:56.997577 2960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:39:56.999964 2960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a: Sending DeleteTablet for 3 replicas of tablet 1d18fa3f641f4bfb86f6b1961146f2a9
I20260926 13:39:57.000779 3097 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d18fa3f641f4bfb86f6b1961146f2a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:56 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:50154
I20260926 13:39:57.000900 3237 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d18fa3f641f4bfb86f6b1961146f2a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:56 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:44302
I20260926 13:39:57.001010 3550 tablet_replica.cc:330] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56: stopping tablet replica
I20260926 13:39:57.001152 3550 raft_consensus.cc:2229] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:39:57.001154 3552 tablet_replica.cc:330] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455: stopping tablet replica
I20260926 13:39:57.001303 3552 raft_consensus.cc:2229] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:39:57.001370 3550 raft_consensus.cc:2258] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:57.001391 3377 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d18fa3f641f4bfb86f6b1961146f2a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:56 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:59942
I20260926 13:39:57.001435 3552 raft_consensus.cc:2258] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:57.001591 3551 tablet_replica.cc:330] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f: stopping tablet replica
I20260926 13:39:57.001729 3551 raft_consensus.cc:2229] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:39:57.001866 3551 raft_consensus.cc:2258] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:57.001772 2960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:39:57.002298 2960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:39:57.002305 3550 ts_tablet_manager.cc:1905] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:57.002658 3551 ts_tablet_manager.cc:1905] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:57.003860 3552 ts_tablet_manager.cc:1905] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:57.005234 3551 ts_tablet_manager.cc:1918] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:39:57.005338 3551 log.cc:1196] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/1d18fa3f641f4bfb86f6b1961146f2a9
I20260926 13:39:57.005321 3550 ts_tablet_manager.cc:1918] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:39:57.005409 3550 log.cc:1196] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-0/wal/wals/1d18fa3f641f4bfb86f6b1961146f2a9
I20260926 13:39:57.005632 3551 ts_tablet_manager.cc:1939] T 1d18fa3f641f4bfb86f6b1961146f2a9 P 2d7528e78bb44af48e8b35fc56e4a24f: Deleting consensus metadata
I20260926 13:39:57.005719 3550 ts_tablet_manager.cc:1939] T 1d18fa3f641f4bfb86f6b1961146f2a9 P fbaec8f354c14a08991df07d4ed10b56: Deleting consensus metadata
I20260926 13:39:57.006554 2944 catalog_manager.cc:4841] TS 2d7528e78bb44af48e8b35fc56e4a24f (127.1.49.3:40027): tablet 1d18fa3f641f4bfb86f6b1961146f2a9 (table test-table [id=aa1603abd418428db35efc674bc578f7]) successfully deleted
I20260926 13:39:57.006778 2944 catalog_manager.cc:4841] TS fbaec8f354c14a08991df07d4ed10b56 (127.1.49.1:35635): tablet 1d18fa3f641f4bfb86f6b1961146f2a9 (table test-table [id=aa1603abd418428db35efc674bc578f7]) successfully deleted
I20260926 13:39:57.006799 3552 ts_tablet_manager.cc:1918] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:39:57.006873 3552 log.cc:1196] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/1d18fa3f641f4bfb86f6b1961146f2a9
I20260926 13:39:57.007170 3552 ts_tablet_manager.cc:1939] T 1d18fa3f641f4bfb86f6b1961146f2a9 P cf1fc9ec2b1448c98557061456f56455: Deleting consensus metadata
I20260926 13:39:57.008085 2945 catalog_manager.cc:4841] TS cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111): tablet 1d18fa3f641f4bfb86f6b1961146f2a9 (table test-table [id=aa1603abd418428db35efc674bc578f7]) successfully deleted
I20260926 13:39:57.011061 3237 tablet_service.cc:1467] Processing CreateTablet for tablet f7972d3b8ff34e529028dfcfe2e07f94 (DEFAULT_TABLE table=test-table [id=ea137bf96c92466c9b809736cfcaa1a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:57.011116 3097 tablet_service.cc:1467] Processing CreateTablet for tablet f7972d3b8ff34e529028dfcfe2e07f94 (DEFAULT_TABLE table=test-table [id=ea137bf96c92466c9b809736cfcaa1a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:57.011179 3377 tablet_service.cc:1467] Processing CreateTablet for tablet f7972d3b8ff34e529028dfcfe2e07f94 (DEFAULT_TABLE table=test-table [id=ea137bf96c92466c9b809736cfcaa1a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:39:57.011400 3097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7972d3b8ff34e529028dfcfe2e07f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:57.011399 3237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7972d3b8ff34e529028dfcfe2e07f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:57.011567 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7972d3b8ff34e529028dfcfe2e07f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:39:57.013900 3561 tablet_bootstrap.cc:492] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f: Bootstrap starting.
I20260926 13:39:57.013955 3564 tablet_bootstrap.cc:492] T f7972d3b8ff34e529028dfcfe2e07f94 P fbaec8f354c14a08991df07d4ed10b56: Bootstrap starting.
I20260926 13:39:57.014009 3562 tablet_bootstrap.cc:492] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455: Bootstrap starting.
I20260926 13:39:57.014868 3561 tablet_bootstrap.cc:654] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:57.014935 3564 tablet_bootstrap.cc:654] T f7972d3b8ff34e529028dfcfe2e07f94 P fbaec8f354c14a08991df07d4ed10b56: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:57.014935 3562 tablet_bootstrap.cc:654] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455: Neither blocks nor log segments found. Creating new log.
I20260926 13:39:57.016424 3564 tablet_bootstrap.cc:492] T f7972d3b8ff34e529028dfcfe2e07f94 P fbaec8f354c14a08991df07d4ed10b56: No bootstrap required, opened a new log
I20260926 13:39:57.016531 3564 ts_tablet_manager.cc:1397] T f7972d3b8ff34e529028dfcfe2e07f94 P fbaec8f354c14a08991df07d4ed10b56: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 13:39:57.017023 3561 tablet_bootstrap.cc:492] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f: No bootstrap required, opened a new log
I20260926 13:39:57.016927 3564 raft_consensus.cc:348] T f7972d3b8ff34e529028dfcfe2e07f94 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:57.017071 3564 raft_consensus.cc:374] T f7972d3b8ff34e529028dfcfe2e07f94 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:57.017103 3564 raft_consensus.cc:729] T f7972d3b8ff34e529028dfcfe2e07f94 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbaec8f354c14a08991df07d4ed10b56, State: Initialized, Role: FOLLOWER
I20260926 13:39:57.017114 3561 ts_tablet_manager.cc:1397] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:39:57.017248 3564 consensus_queue.cc:260] T f7972d3b8ff34e529028dfcfe2e07f94 P fbaec8f354c14a08991df07d4ed10b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:57.017554 3561 raft_consensus.cc:348] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:57.017673 3561 raft_consensus.cc:374] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:57.017705 3561 raft_consensus.cc:729] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7528e78bb44af48e8b35fc56e4a24f, State: Initialized, Role: FOLLOWER
I20260926 13:39:57.017828 3561 consensus_queue.cc:260] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:57.017593 3564 ts_tablet_manager.cc:1428] T f7972d3b8ff34e529028dfcfe2e07f94 P fbaec8f354c14a08991df07d4ed10b56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:39:57.018126 3561 ts_tablet_manager.cc:1428] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:39:57.028258 3562 tablet_bootstrap.cc:492] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455: No bootstrap required, opened a new log
I20260926 13:39:57.028375 3562 ts_tablet_manager.cc:1397] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260926 13:39:57.029027 3562 raft_consensus.cc:348] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:57.029155 3562 raft_consensus.cc:374] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:39:57.029181 3562 raft_consensus.cc:729] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1fc9ec2b1448c98557061456f56455, State: Initialized, Role: FOLLOWER
I20260926 13:39:57.029366 3562 consensus_queue.cc:260] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:57.029727 3562 ts_tablet_manager.cc:1428] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:39:57.067857 3536 raft_consensus.cc:482] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:39:57.068007 3536 raft_consensus.cc:504] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:57.068398 3536 leader_election.cc:288] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbaec8f354c14a08991df07d4ed10b56 (127.1.49.1:35635), cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111)
I20260926 13:39:57.068941 3117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7972d3b8ff34e529028dfcfe2e07f94" candidate_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbaec8f354c14a08991df07d4ed10b56" is_pre_election: true
I20260926 13:39:57.068953 3257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7972d3b8ff34e529028dfcfe2e07f94" candidate_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1fc9ec2b1448c98557061456f56455" is_pre_election: true
I20260926 13:39:57.069094 3117 raft_consensus.cc:2454] T f7972d3b8ff34e529028dfcfe2e07f94 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d7528e78bb44af48e8b35fc56e4a24f in term 0.
I20260926 13:39:57.069131 3257 raft_consensus.cc:2454] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d7528e78bb44af48e8b35fc56e4a24f in term 0.
I20260926 13:39:57.069514 3329 leader_election.cc:302] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d7528e78bb44af48e8b35fc56e4a24f, cf1fc9ec2b1448c98557061456f56455, fbaec8f354c14a08991df07d4ed10b56; no voters:
I20260926 13:39:57.069732 3536 raft_consensus.cc:2781] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:39:57.069799 3536 raft_consensus.cc:482] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:39:57.069830 3536 raft_consensus.cc:3037] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:57.070967 3536 raft_consensus.cc:504] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:57.071400 3536 leader_election.cc:288] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [CANDIDATE]: Term 1 election: Requested vote from peers fbaec8f354c14a08991df07d4ed10b56 (127.1.49.1:35635), cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111)
I20260926 13:39:57.071874 3117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7972d3b8ff34e529028dfcfe2e07f94" candidate_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbaec8f354c14a08991df07d4ed10b56"
I20260926 13:39:57.071929 3257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7972d3b8ff34e529028dfcfe2e07f94" candidate_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1fc9ec2b1448c98557061456f56455"
I20260926 13:39:57.072039 3117 raft_consensus.cc:3037] T f7972d3b8ff34e529028dfcfe2e07f94 P fbaec8f354c14a08991df07d4ed10b56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:57.072125 3257 raft_consensus.cc:3037] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:39:57.073093 3117 raft_consensus.cc:2454] T f7972d3b8ff34e529028dfcfe2e07f94 P fbaec8f354c14a08991df07d4ed10b56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d7528e78bb44af48e8b35fc56e4a24f in term 1.
I20260926 13:39:57.073136 3257 raft_consensus.cc:2454] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d7528e78bb44af48e8b35fc56e4a24f in term 1.
I20260926 13:39:57.073544 3328 leader_election.cc:302] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d7528e78bb44af48e8b35fc56e4a24f, cf1fc9ec2b1448c98557061456f56455, fbaec8f354c14a08991df07d4ed10b56; no voters:
I20260926 13:39:57.073720 3536 raft_consensus.cc:2781] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:39:57.073843 3536 raft_consensus.cc:686] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 LEADER]: Becoming Leader. State: Replica: 2d7528e78bb44af48e8b35fc56e4a24f, State: Running, Role: LEADER
I20260926 13:39:57.073962 3536 consensus_queue.cc:237] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } }
I20260926 13:39:57.075973 2960 catalog_manager.cc:5495] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d7528e78bb44af48e8b35fc56e4a24f (127.1.49.3). New cstate: current_term: 1 leader_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d7528e78bb44af48e8b35fc56e4a24f" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40027 } health_report { overall_health: HEALTHY } } }
I20260926 13:39:57.104640 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:57.100186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.1.49.3:40027)
0926 13:39:57.100521 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 13:39:57.100529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:39:57.100543 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:39:57.100605 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:39:57.100609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:39:57.100658 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:39:57.100740 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:39:57.100999 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:57.101689 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:57.103056 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:39:57.103300 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:39:57.103399 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:39:57.103889 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:39:57.103986 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:39:57.104143 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 13:39:57.104421 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 13:39:57.104490 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":161,"threads_started":1}
I20260926 13:39:57.112516 3257 raft_consensus.cc:1261] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Refusing update from remote peer 2d7528e78bb44af48e8b35fc56e4a24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:57.112529 3117 raft_consensus.cc:1261] T f7972d3b8ff34e529028dfcfe2e07f94 P fbaec8f354c14a08991df07d4ed10b56 [term 1 FOLLOWER]: Refusing update from remote peer 2d7528e78bb44af48e8b35fc56e4a24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:39:57.113003 3505 consensus_queue.cc:1035] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1fc9ec2b1448c98557061456f56455" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:39:57.113191 3536 consensus_queue.cc:1035] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbaec8f354c14a08991df07d4ed10b56" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 13:39:57.129315 3529 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 13:39:58.699752 2959 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:39:58.700021 2959 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:39:58.702997 2959 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 241452488bed4fb1ad4969113583b95a: Sending DeleteTablet for 3 replicas of tablet f7972d3b8ff34e529028dfcfe2e07f94
I20260926 13:39:58.703755 3097 tablet_service.cc:1514] Processing DeleteTablet for tablet f7972d3b8ff34e529028dfcfe2e07f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:58 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:50154
I20260926 13:39:58.703794 3377 tablet_service.cc:1514] Processing DeleteTablet for tablet f7972d3b8ff34e529028dfcfe2e07f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:58 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:59942
I20260926 13:39:58.703819 3237 tablet_service.cc:1514] Processing DeleteTablet for tablet f7972d3b8ff34e529028dfcfe2e07f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:39:58 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:44302
I20260926 13:39:58.704214 3602 tablet_replica.cc:330] T f7972d3b8ff34e529028dfcfe2e07f94 P fbaec8f354c14a08991df07d4ed10b56: stopping tablet replica
I20260926 13:39:58.704214 3604 tablet_replica.cc:330] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f: stopping tablet replica
I20260926 13:39:58.704241 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 3032
I20260926 13:39:58.705987 3603 tablet_replica.cc:330] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455: stopping tablet replica
I20260926 13:39:58.706177 3603 raft_consensus.cc:2229] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:39:58.706028 3604 raft_consensus.cc:2229] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:39:58.706341 3603 raft_consensus.cc:2258] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:58.706430 3604 raft_consensus.cc:2258] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:39:58.707165 3604 ts_tablet_manager.cc:1905] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:58.707993 3603 ts_tablet_manager.cc:1905] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:39:58.710316 3604 ts_tablet_manager.cc:1918] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:39:58.710415 3604 log.cc:1196] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/f7972d3b8ff34e529028dfcfe2e07f94
I20260926 13:39:58.710743 3604 ts_tablet_manager.cc:1939] T f7972d3b8ff34e529028dfcfe2e07f94 P 2d7528e78bb44af48e8b35fc56e4a24f: Deleting consensus metadata
I20260926 13:39:58.710779 3603 ts_tablet_manager.cc:1918] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:39:58.710848 3603 log.cc:1196] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/f7972d3b8ff34e529028dfcfe2e07f94
I20260926 13:39:58.711180 3603 ts_tablet_manager.cc:1939] T f7972d3b8ff34e529028dfcfe2e07f94 P cf1fc9ec2b1448c98557061456f56455: Deleting consensus metadata
I20260926 13:39:58.711933 2944 catalog_manager.cc:4841] TS 2d7528e78bb44af48e8b35fc56e4a24f (127.1.49.3:40027): tablet f7972d3b8ff34e529028dfcfe2e07f94 (table test-table [id=ea137bf96c92466c9b809736cfcaa1a1]) successfully deleted
I20260926 13:39:58.712200 2945 catalog_manager.cc:4841] TS cf1fc9ec2b1448c98557061456f56455 (127.1.49.2:36111): tablet f7972d3b8ff34e529028dfcfe2e07f94 (table test-table [id=ea137bf96c92466c9b809736cfcaa1a1]) successfully deleted
I20260926 13:39:58.712693 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 3172
W20260926 13:39:58.712724 2944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:35635 (error 108)
I20260926 13:39:58.715350 3606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:39:58.714750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:35635 (local address 127.0.0.1:50188)
0926 13:39:58.715075 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 13:39:58.715185 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:35635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
W20260926 13:39:58.715536 2944 catalog_manager.cc:4584] TS fbaec8f354c14a08991df07d4ed10b56 (127.1.49.1:35635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7972d3b8ff34e529028dfcfe2e07f94: Network error: Client connection negotiation failed: client connection to 127.1.49.1:35635: connect: Connection refused (error 111)
I20260926 13:39:58.719779 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 3312
I20260926 13:39:58.727546 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 2928
2026-09-26T13:39:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6736 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:39:58 dist-test-slave-1wns krb5kdc[3609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:39:58 dist-test-slave-1wns krb5kdc[3609](info): set up 2 sockets
Sep 26 13:39:58 dist-test-slave-1wns krb5kdc[3609](info): commencing operation
krb5kdc: starting...
W20260926 13:39:59.828088 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:39:59 dist-test-slave-1wns krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:39:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:39:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:39:59.964318 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:39303
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:45739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:39303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:00.085381 3622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:00.085707 3622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:00.085783 3622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:00.090157 3622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:00.090272 3622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:00.090310 3622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:00.090366 3622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:00.090420 3622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:00.096127 3622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:45739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:39303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:39303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:00.097330 3622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:00.098448 3622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:40:00.107836 3622 server_base.cc:1034] running on GCE node
W20260926 13:40:00.107780 3628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:00.107774 3630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:00.107774 3627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:00.108675 3622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:00.109956 3622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:00.111172 3622 hybrid_clock.cc:648] HybridClock initialized: now 1790430000111155 us; error 36 us; skew 500 ppm
Sep 26 13:40:00 dist-test-slave-1wns krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430000, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:00.114475 3622 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:40:00.115716 3622 webserver.cc:460] Webserver started at http://127.1.49.62:33113/ using document root <none> and password file <none>
I20260926 13:40:00.116428 3622 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:00.116542 3622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:00.116820 3622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:00.119194 3622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "cc0741db33844e90a68a9733ef9e7cf1"
format_stamp: "Formatted at 2026-09-26 13:40:00 on dist-test-slave-1wns"
server_key: "4cf9ebd9e9e40843ff1aa6ae543e0778"
server_key_iv: "d6a5d0ab40eb9e450fde06661fc21685"
server_key_version: "encryptionkey@0"
I20260926 13:40:00.119990 3622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "cc0741db33844e90a68a9733ef9e7cf1"
format_stamp: "Formatted at 2026-09-26 13:40:00 on dist-test-slave-1wns"
server_key: "4cf9ebd9e9e40843ff1aa6ae543e0778"
server_key_iv: "d6a5d0ab40eb9e450fde06661fc21685"
server_key_version: "encryptionkey@0"
I20260926 13:40:00.124554 3622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:40:00.127452 3637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:00.129101 3622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:00.129309 3622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "cc0741db33844e90a68a9733ef9e7cf1"
format_stamp: "Formatted at 2026-09-26 13:40:00 on dist-test-slave-1wns"
server_key: "4cf9ebd9e9e40843ff1aa6ae543e0778"
server_key_iv: "d6a5d0ab40eb9e450fde06661fc21685"
server_key_version: "encryptionkey@0"
I20260926 13:40:00.129465 3622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:00.162804 3622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:00.167104 3622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:00.167384 3622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:00.176693 3622 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:39303
I20260926 13:40:00.176719 3689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:39303 every 8 connection(s)
I20260926 13:40:00.177875 3622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:00.181252 3690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:00.182662 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 3622
I20260926 13:40:00.182782 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:00.183080 1220 external_mini_cluster.cc:1428] Setting key 66d3c1f3c3ce2269d5308c847e142d52
I20260926 13:40:00.188277 3690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1: Bootstrap starting.
I20260926 13:40:00.191725 3690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:00.192947 3690 log.cc:824] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1: Log is configured to *not* fsync() on all Append() calls
Sep 26 13:40:00 dist-test-slave-1wns krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790429999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:00.195765 3690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1: No bootstrap required, opened a new log
I20260926 13:40:00.200201 3690 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0741db33844e90a68a9733ef9e7cf1" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 39303 } }
I20260926 13:40:00.200495 3690 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:00.200601 3690 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0741db33844e90a68a9733ef9e7cf1, State: Initialized, Role: FOLLOWER
I20260926 13:40:00.200613 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:00.184705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.1.49.62:39303)
0926 13:40:00.185144 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:00.185176 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:00.185214 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:40:00.185997 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:00.186017 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:00.186446 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:00.186805 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:00.187914 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:00.189304 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:00.190706 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:00.191244 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:00.191379 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:00.194878 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:00.194884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:00.194901 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:00.194930 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:00.197106 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:00.197620 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:00.197627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:00.197634 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:00.197779 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:00.198085 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:00.198089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:00.198090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:00.198640 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:00.198911 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:00.199193 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 13:40:00.199523 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":155,"threads_started":1}
I20260926 13:40:00.201215 3690 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0741db33844e90a68a9733ef9e7cf1" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 39303 } }
I20260926 13:40:00.201387 3690 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:00.201476 3690 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:00.201581 3690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:00.206354 3690 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0741db33844e90a68a9733ef9e7cf1" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 39303 } }
I20260926 13:40:00.206874 3690 leader_election.cc:302] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc0741db33844e90a68a9733ef9e7cf1; no voters:
I20260926 13:40:00.207393 3690 leader_election.cc:288] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:00.207646 3695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:00.207942 3695 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [term 1 LEADER]: Becoming Leader. State: Replica: cc0741db33844e90a68a9733ef9e7cf1, State: Running, Role: LEADER
I20260926 13:40:00.208381 3695 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0741db33844e90a68a9733ef9e7cf1" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 39303 } }
I20260926 13:40:00.208994 3690 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:00.209884 3695 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc0741db33844e90a68a9733ef9e7cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0741db33844e90a68a9733ef9e7cf1" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 39303 } } }
I20260926 13:40:00.210157 3695 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:00.210485 3695 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc0741db33844e90a68a9733ef9e7cf1. Latest consensus state: current_term: 1 leader_uuid: "cc0741db33844e90a68a9733ef9e7cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc0741db33844e90a68a9733ef9e7cf1" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 39303 } } }
I20260926 13:40:00.210672 3695 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:00.211037 3704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:00.214915 3704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:00.222806 3704 catalog_manager.cc:1347] Generated new cluster ID: 3289de0ecff7477f9ffe26f3b97eed69
I20260926 13:40:00.222911 3704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:00.234839 3704 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:00.236335 3704 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:00.247054 3704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc0741db33844e90a68a9733ef9e7cf1: Generated new TSK 0
I20260926 13:40:00.248113 3704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
W20260926 13:40:01.390923 1220 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.49.1: real 1.132s user 0.000s sys 0.004s
I20260926 13:40:01.391454 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:39303
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:45739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:01.527251 3716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:01.527683 3716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:01.527853 3716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:01.532497 3716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:01.532626 3716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:01.532742 3716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:40:01.539245 3716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:45739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:39303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:01.540562 3716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:01.542047 3716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:01.553375 3724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:01.553375 3721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:01.553375 3722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:01.553785 3716 server_base.cc:1034] running on GCE node
I20260926 13:40:01.554319 3716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:01.555076 3716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:01.556294 3716 hybrid_clock.cc:648] HybridClock initialized: now 1790430001556271 us; error 38 us; skew 500 ppm
Sep 26 13:40:01 dist-test-slave-1wns krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430001, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:01.559732 3716 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:40:01.560873 3716 webserver.cc:460] Webserver started at http://127.1.49.1:46105/ using document root <none> and password file <none>
I20260926 13:40:01.561494 3716 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:01.561594 3716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:01.561842 3716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:01.564621 3716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "af5e6ef6e01842708f8a4ebe4f14d84d"
format_stamp: "Formatted at 2026-09-26 13:40:01 on dist-test-slave-1wns"
server_key: "2181834122ed6b861e955d1af25c9cab"
server_key_iv: "4f6f32799a39a480d26897edb8924708"
server_key_version: "encryptionkey@0"
I20260926 13:40:01.565277 3716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "af5e6ef6e01842708f8a4ebe4f14d84d"
format_stamp: "Formatted at 2026-09-26 13:40:01 on dist-test-slave-1wns"
server_key: "2181834122ed6b861e955d1af25c9cab"
server_key_iv: "4f6f32799a39a480d26897edb8924708"
server_key_version: "encryptionkey@0"
I20260926 13:40:01.570112 3716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 13:40:01.573262 3731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:01.575096 3716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:40:01.575347 3716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "af5e6ef6e01842708f8a4ebe4f14d84d"
format_stamp: "Formatted at 2026-09-26 13:40:01 on dist-test-slave-1wns"
server_key: "2181834122ed6b861e955d1af25c9cab"
server_key_iv: "4f6f32799a39a480d26897edb8924708"
server_key_version: "encryptionkey@0"
I20260926 13:40:01.575495 3716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:01.595299 3716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:01.599506 3716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:01.599754 3716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:01.600471 3716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:01.601516 3716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:01.601605 3716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:01.601689 3716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:01.601747 3716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:01.613389 3716 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:45415
I20260926 13:40:01.613441 3844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:45415 every 8 connection(s)
I20260926 13:40:01.614524 3716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:40:01.620247 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 3716
I20260926 13:40:01.620385 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:40:01.620673 1220 external_mini_cluster.cc:1428] Setting key 0baba96b08c741ac34bf7730d876b681
Sep 26 13:40:01 dist-test-slave-1wns krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430001, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
I20260926 13:40:01.632067 3848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:01.617470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:39303 (local address 127.1.49.1:49501)
0926 13:40:01.618029 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:01.618078 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:01.619422 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:01.620194 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:01.620206 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:01.620758 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:01.621501 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:01.621525 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:01.622365 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:01.622369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:01.623352 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:01.623361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:01.623916 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:01.623920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:01.624113 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:01.624826 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:01.624844 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:01.626454 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:01.628794 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:01.628803 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:01.628823 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:01.629147 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:01.629503 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:01.629506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:01.629508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:01.629678 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:01.630151 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:01.630160 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:01.630468 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 13:40:01.630723 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 13:40:01.631060 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":145,"threads_started":1}
I20260926 13:40:01.632313 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:01.616913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:49501 (local address 127.1.49.62:39303)
0926 13:40:01.617220 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:01.617226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:01.618215 (+ 989us) server_negotiation.cc:407] Connection header received
0926 13:40:01.619687 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:01.619690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:01.619754 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:01.619893 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:01.621676 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:01.622229 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:01.623489 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:01.623732 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:01.623874 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:01.626656 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:01.626660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:01.626662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:01.626688 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:01.628631 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:01.629276 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:01.629280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:01.629283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:01.629382 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:01.629788 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:01.629794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:01.629796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:01.630023 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:01.630137 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:01.631969 (+ 1832us) server_negotiation.cc:299] Negotiation successful
0926 13:40:01.632130 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
I20260926 13:40:01.633659 3845 heartbeater.cc:347] Connected to a master server at 127.1.49.62:39303
I20260926 13:40:01.634003 3845 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:01.634732 3845 heartbeater.cc:511] Master 127.1.49.62:39303 requested a full tablet report, sending...
I20260926 13:40:01.636672 3654 ts_manager.cc:194] Registered new tserver with Master: af5e6ef6e01842708f8a4ebe4f14d84d (127.1.49.1:45415)
I20260926 13:40:01.637871 3654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:49501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
I20260926 13:40:02.640921 3845 heartbeater.cc:503] Master 127.1.49.62:39303 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
W20260926 13:40:02.973587 1220 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.49.2: real 1.351s user 0.002s sys 0.002s
I20260926 13:40:02.974166 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:39303
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:45739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:03.104380 3851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:03.104813 3851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:03.104909 3851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:03.109589 3851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:03.109692 3851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:03.109798 3851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:40:03.115998 3851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:45739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:39303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:03.117316 3851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:03.118497 3851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:40:03.129550 3851 server_base.cc:1034] running on GCE node
W20260926 13:40:03.129499 3856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:03.129489 3859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:03.129489 3857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:03.130524 3851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:03.131405 3851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:03.132632 3851 hybrid_clock.cc:648] HybridClock initialized: now 1790430003132587 us; error 60 us; skew 500 ppm
Sep 26 13:40:03 dist-test-slave-1wns krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430003, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:03.136209 3851 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:40:03.137327 3851 webserver.cc:460] Webserver started at http://127.1.49.2:38159/ using document root <none> and password file <none>
I20260926 13:40:03.137979 3851 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:03.138077 3851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:03.138324 3851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:03.140897 3851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "25b33044e2ad4998a61611ee4bed16f6"
format_stamp: "Formatted at 2026-09-26 13:40:03 on dist-test-slave-1wns"
server_key: "89525a2626d1b33cd7bd2f03b2d04bcd"
server_key_iv: "48338cdedd6b77e588ba78691ed241be"
server_key_version: "encryptionkey@0"
I20260926 13:40:03.141717 3851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "25b33044e2ad4998a61611ee4bed16f6"
format_stamp: "Formatted at 2026-09-26 13:40:03 on dist-test-slave-1wns"
server_key: "89525a2626d1b33cd7bd2f03b2d04bcd"
server_key_iv: "48338cdedd6b77e588ba78691ed241be"
server_key_version: "encryptionkey@0"
I20260926 13:40:03.146358 3851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:40:03.149489 3866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:03.151081 3851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:40:03.151255 3851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "25b33044e2ad4998a61611ee4bed16f6"
format_stamp: "Formatted at 2026-09-26 13:40:03 on dist-test-slave-1wns"
server_key: "89525a2626d1b33cd7bd2f03b2d04bcd"
server_key_iv: "48338cdedd6b77e588ba78691ed241be"
server_key_version: "encryptionkey@0"
I20260926 13:40:03.151419 3851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:03.170600 3851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:03.174496 3851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:03.174736 3851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:03.175406 3851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:03.176407 3851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:03.176496 3851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:03.176604 3851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:03.176658 3851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:03.187434 3851 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:42751
I20260926 13:40:03.187462 3979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:42751 every 8 connection(s)
I20260926 13:40:03.188493 3851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:40:03.192276 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 3851
I20260926 13:40:03.192373 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:40:03.192692 1220 external_mini_cluster.cc:1428] Setting key a378700c0cfb9916fd97052998fa61e7
Sep 26 13:40:03 dist-test-slave-1wns krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430003, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
I20260926 13:40:03.204596 3982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:03.190762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:55813 (local address 127.1.49.62:39303)
0926 13:40:03.191039 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:03.191045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:03.191879 (+ 834us) server_negotiation.cc:407] Connection header received
0926 13:40:03.193530 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:03.193533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:03.193590 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:03.193677 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:03.195185 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:03.195734 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:03.197075 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:03.197318 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:03.197437 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:03.200421 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:03.200425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:03.200427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:03.200454 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:03.202226 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:03.202839 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:03.202843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:03.202845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:03.202935 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:03.203307 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:03.203311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:03.203313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:03.203519 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:03.203630 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:03.204258 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 13:40:03.204401 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":59,"threads_started":1}
I20260926 13:40:03.205456 3983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:03.191195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:39303 (local address 127.1.49.2:55813)
0926 13:40:03.191691 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:03.191737 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:03.193299 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:03.193896 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:03.193905 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:03.194486 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:03.195019 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:03.195033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:03.195885 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:03.195889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:03.196940 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:03.196956 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:03.197479 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:03.197484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:03.197677 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:03.198340 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:03.198357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:03.200218 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:03.202377 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:03.202385 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:03.202405 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:03.202718 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:03.203030 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:03.203034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:03.203036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:03.203204 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:03.203643 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:03.203653 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:03.203997 (+ 344us) client_negotiation.cc:769] Sending connection context
0926 13:40:03.204204 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 13:40:03.204571 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":122,"threads_started":1}
I20260926 13:40:03.206854 3980 heartbeater.cc:347] Connected to a master server at 127.1.49.62:39303
I20260926 13:40:03.207110 3980 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:03.207630 3980 heartbeater.cc:511] Master 127.1.49.62:39303 requested a full tablet report, sending...
I20260926 13:40:03.208907 3654 ts_manager.cc:194] Registered new tserver with Master: 25b33044e2ad4998a61611ee4bed16f6 (127.1.49.2:42751)
I20260926 13:40:03.209940 3654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:55813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
I20260926 13:40:04.212481 3980 heartbeater.cc:503] Master 127.1.49.62:39303 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
W20260926 13:40:04.253796 1220 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.49.3: real 1.059s user 0.000s sys 0.004s
I20260926 13:40:04.254468 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:39303
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:45739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:04.384733 3986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:04.385061 3986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:04.385157 3986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:04.389827 3986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:04.389915 3986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:04.389979 3986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:40:04.395682 3986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:45739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:39303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:04.396924 3986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:04.397964 3986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:04.408812 3992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:04.408811 3991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:04.408954 3986 server_base.cc:1034] running on GCE node
W20260926 13:40:04.408829 3994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:04.409693 3986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:04.410544 3986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:04.411762 3986 hybrid_clock.cc:648] HybridClock initialized: now 1790430004411716 us; error 61 us; skew 500 ppm
Sep 26 13:40:04 dist-test-slave-1wns krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430004, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:04.423231 3986 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:40:04.424577 3986 webserver.cc:460] Webserver started at http://127.1.49.3:37311/ using document root <none> and password file <none>
I20260926 13:40:04.425288 3986 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:04.425396 3986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:04.425684 3986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:04.428215 3986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "fa1877d92b944c8d8d05c5dd5ea316c5"
format_stamp: "Formatted at 2026-09-26 13:40:04 on dist-test-slave-1wns"
server_key: "5830aa30abdd17b2858326a3d2009b66"
server_key_iv: "b22501e04fbccf94c6bfeb7fdcaeb7c2"
server_key_version: "encryptionkey@0"
I20260926 13:40:04.428894 3986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "fa1877d92b944c8d8d05c5dd5ea316c5"
format_stamp: "Formatted at 2026-09-26 13:40:04 on dist-test-slave-1wns"
server_key: "5830aa30abdd17b2858326a3d2009b66"
server_key_iv: "b22501e04fbccf94c6bfeb7fdcaeb7c2"
server_key_version: "encryptionkey@0"
I20260926 13:40:04.433358 3986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:40:04.436275 4001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:04.437698 3986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:04.437906 3986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "fa1877d92b944c8d8d05c5dd5ea316c5"
format_stamp: "Formatted at 2026-09-26 13:40:04 on dist-test-slave-1wns"
server_key: "5830aa30abdd17b2858326a3d2009b66"
server_key_iv: "b22501e04fbccf94c6bfeb7fdcaeb7c2"
server_key_version: "encryptionkey@0"
I20260926 13:40:04.438076 3986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:04.452960 3986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:04.456890 3986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:04.457129 3986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:04.457805 3986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:04.458760 3986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:04.458839 3986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:04.458926 3986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:04.458976 3986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:04.469673 3986 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:34693
I20260926 13:40:04.469698 4114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:34693 every 8 connection(s)
I20260926 13:40:04.470734 3986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:40:04.472692 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 3986
I20260926 13:40:04.472827 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:40:04.473174 1220 external_mini_cluster.cc:1428] Setting key 721a801a81f73d98afa90c89f82ab14c
Sep 26 13:40:04 dist-test-slave-1wns krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430004, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:04.487098 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:04.473092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:54677 (local address 127.1.49.62:39303)
0926 13:40:04.473355 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:04.473359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:04.474557 (+ 1198us) server_negotiation.cc:407] Connection header received
0926 13:40:04.476032 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:04.476035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:04.476083 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:04.476160 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:04.477459 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:04.478024 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:04.479357 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:04.479611 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:04.479745 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:04.482723 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:04.482727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:04.482729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:04.482758 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:04.484677 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:04.485234 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:04.485237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:04.485239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:04.485324 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:04.485643 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:04.485645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:04.485647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:04.485847 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:04.485931 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:04.486704 (+ 773us) server_negotiation.cc:299] Negotiation successful
0926 13:40:04.486866 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":97,"threads_started":1}
I20260926 13:40:04.488204 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:04.473732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:39303 (local address 127.1.49.3:54677)
0926 13:40:04.474386 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:04.474444 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:04.475772 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:04.476354 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:04.476362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:04.476764 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:04.477295 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:04.477309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:04.478173 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:04.478176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:04.479158 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:04.479169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:04.479793 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:04.479798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:04.480058 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:04.480841 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:04.480859 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:04.482500 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:04.484801 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:04.484806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:04.484821 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:04.485128 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:04.485410 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:04.485412 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:04.485413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:04.485548 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:04.485961 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:04.485968 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:04.486342 (+ 374us) client_negotiation.cc:769] Sending connection context
0926 13:40:04.486581 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 13:40:04.486955 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":181,"threads_started":1}
I20260926 13:40:04.489444 4115 heartbeater.cc:347] Connected to a master server at 127.1.49.62:39303
I20260926 13:40:04.489712 4115 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:04.490243 4115 heartbeater.cc:511] Master 127.1.49.62:39303 requested a full tablet report, sending...
I20260926 13:40:04.491499 3654 ts_manager.cc:194] Registered new tserver with Master: fa1877d92b944c8d8d05c5dd5ea316c5 (127.1.49.3:34693)
I20260926 13:40:04.492200 3654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:54677
I20260926 13:40:04.500670 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 13:40:04.515476 4117 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:04.513919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.1.49.62:39303)
0926 13:40:04.514058 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:04.514063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:04.514251 (+ 188us) server_negotiation.cc:407] Connection header received
0926 13:40:04.514422 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:04.514426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:04.514478 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:04.514571 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:04.515360 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58188: BlockingRecv error: recv got EOF from 127.0.0.1:58188 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:40:04.516741 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 3716
I20260926 13:40:04.522737 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 3851
I20260926 13:40:04.528699 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 3986
I20260926 13:40:04.534176 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 3622
2026-09-26T13:40:04Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5804 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:40:04 dist-test-slave-1wns krb5kdc[4128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:40:04 dist-test-slave-1wns krb5kdc[4128](info): set up 2 sockets
Sep 26 13:40:04 dist-test-slave-1wns krb5kdc[4128](info): commencing operation
krb5kdc: starting...
W20260926 13:40:05.632141 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:40:05 dist-test-slave-1wns krb5kdc[4128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:40:05.772573 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:33873
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:43233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:33873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:05.896378 4141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:05.896739 4141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:05.896812 4141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:05.901055 4141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:05.901166 4141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:05.901204 4141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:05.901242 4141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:05.901350 4141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:05.908331 4141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:43233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:33873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:33873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:05.909442 4141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:05.910502 4141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:40:05.919724 4141 server_base.cc:1034] running on GCE node
W20260926 13:40:05.919652 4147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:05.919684 4146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:05.919663 4149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:05.920671 4141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:05.922036 4141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:05.923266 4141 hybrid_clock.cc:648] HybridClock initialized: now 1790430005923246 us; error 36 us; skew 500 ppm
Sep 26 13:40:05 dist-test-slave-1wns krb5kdc[4128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430005, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:05.926779 4141 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:40:05.927963 4141 webserver.cc:460] Webserver started at http://127.1.49.62:37923/ using document root <none> and password file <none>
I20260926 13:40:05.928581 4141 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:05.928651 4141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:05.928836 4141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:05.931144 4141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "a736d3fdc3394ada9d01df4e34495654"
format_stamp: "Formatted at 2026-09-26 13:40:05 on dist-test-slave-1wns"
server_key: "adc53c3b21d9dfbd2693dcb7299d2d8c"
server_key_iv: "32f0b71595b398a05beeb86001eb2fba"
server_key_version: "encryptionkey@0"
I20260926 13:40:05.931731 4141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "a736d3fdc3394ada9d01df4e34495654"
format_stamp: "Formatted at 2026-09-26 13:40:05 on dist-test-slave-1wns"
server_key: "adc53c3b21d9dfbd2693dcb7299d2d8c"
server_key_iv: "32f0b71595b398a05beeb86001eb2fba"
server_key_version: "encryptionkey@0"
I20260926 13:40:05.936072 4141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 13:40:05.939116 4156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:05.940711 4141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:05.940860 4141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "a736d3fdc3394ada9d01df4e34495654"
format_stamp: "Formatted at 2026-09-26 13:40:05 on dist-test-slave-1wns"
server_key: "adc53c3b21d9dfbd2693dcb7299d2d8c"
server_key_iv: "32f0b71595b398a05beeb86001eb2fba"
server_key_version: "encryptionkey@0"
I20260926 13:40:05.940987 4141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:05.960739 4141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:05.964524 4141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:05.964757 4141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:05.972501 4141 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:33873
I20260926 13:40:05.972858 4208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:33873 every 8 connection(s)
I20260926 13:40:05.973624 4141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:05.976796 4209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:05.980183 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 4141
I20260926 13:40:05.980321 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:05.980578 1220 external_mini_cluster.cc:1428] Setting key 87ef16110bf3f5970cb9f69d03b707a6
I20260926 13:40:05.984321 4209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654: Bootstrap starting.
I20260926 13:40:05.987196 4209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:05.988154 4209 log.cc:824] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:05.990357 4209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654: No bootstrap required, opened a new log
Sep 26 13:40:05 dist-test-slave-1wns krb5kdc[4128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:05.994175 4209 raft_consensus.cc:348] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a736d3fdc3394ada9d01df4e34495654" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33873 } }
I20260926 13:40:05.994459 4209 raft_consensus.cc:374] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:05.994561 4209 raft_consensus.cc:729] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a736d3fdc3394ada9d01df4e34495654, State: Initialized, Role: FOLLOWER
I20260926 13:40:05.995123 4209 consensus_queue.cc:260] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a736d3fdc3394ada9d01df4e34495654" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33873 } }
I20260926 13:40:05.995370 4209 raft_consensus.cc:388] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:05.995466 4209 raft_consensus.cc:482] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:05.995610 4209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:05.996726 4209 raft_consensus.cc:504] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a736d3fdc3394ada9d01df4e34495654" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33873 } }
I20260926 13:40:05.997157 4209 leader_election.cc:302] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a736d3fdc3394ada9d01df4e34495654; no voters:
I20260926 13:40:05.997591 4209 leader_election.cc:288] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:05.997711 4214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:05.997925 4214 raft_consensus.cc:686] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [term 1 LEADER]: Becoming Leader. State: Replica: a736d3fdc3394ada9d01df4e34495654, State: Running, Role: LEADER
I20260926 13:40:05.998001 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:05.982068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.1.49.62:33873)
0926 13:40:05.982530 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:05.982549 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:05.982607 (+ 58us) server_negotiation.cc:407] Connection header received
0926 13:40:05.983465 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:05.983493 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:05.984024 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:05.984433 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:05.985585 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:05.987076 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:05.988264 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:05.988891 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:05.989091 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:05.991672 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:05.991679 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:05.991693 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:05.991722 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:05.994232 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:05.994841 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:05.994848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:05.994853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:05.994979 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:05.995288 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:05.995294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:05.995296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:05.995970 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:05.996211 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:05.996542 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 13:40:05.996923 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":135,"threads_started":1}
I20260926 13:40:05.998723 4214 consensus_queue.cc:237] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a736d3fdc3394ada9d01df4e34495654" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33873 } }
I20260926 13:40:05.999035 4209 sys_catalog.cc:564] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:06.001137 4216 sys_catalog.cc:455] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a736d3fdc3394ada9d01df4e34495654. Latest consensus state: current_term: 1 leader_uuid: "a736d3fdc3394ada9d01df4e34495654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a736d3fdc3394ada9d01df4e34495654" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33873 } } }
I20260926 13:40:06.001276 4216 sys_catalog.cc:458] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:06.002125 4215 sys_catalog.cc:455] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a736d3fdc3394ada9d01df4e34495654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a736d3fdc3394ada9d01df4e34495654" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33873 } } }
I20260926 13:40:06.002239 4215 sys_catalog.cc:458] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:06.002701 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 13:40:06.004828 4228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:40:06.004989 4228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:40:06.005089 4229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:06.005810 4229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:06.012100 4229 catalog_manager.cc:1347] Generated new cluster ID: caec8bc4714a4485bcca4c5286d221ba
I20260926 13:40:06.012255 4229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:06.028961 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:06.023833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.1.49.62:33873)
0926 13:40:06.024027 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:06.024032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:06.024131 (+ 99us) server_negotiation.cc:407] Connection header received
0926 13:40:06.024315 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:06.024320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:06.024373 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:06.024463 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:06.025020 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:06.025615 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:06.026527 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:06.026788 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:06.026903 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:06.027880 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:06.027884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:06.027887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:06.028410 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:06.028444 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:06.028461 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:06.028577 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:06.028715 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 13:40:06.028791 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":109}
I20260926 13:40:06.040292 4229 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:06.041481 4229 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:06.048436 4229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a736d3fdc3394ada9d01df4e34495654: Generated new TSK 0
I20260926 13:40:06.049450 4229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:40:06.063375 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 4141
2026-09-26T13:40:06Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1526 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:40:06 dist-test-slave-1wns krb5kdc[4242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:40:06 dist-test-slave-1wns krb5kdc[4242](info): set up 2 sockets
Sep 26 13:40:06 dist-test-slave-1wns krb5kdc[4242](info): commencing operation
krb5kdc: starting...
W20260926 13:40:07.144124 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:40:07 dist-test-slave-1wns krb5kdc[4242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:40:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:40:07.276405 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:40263
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:40263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:07.393612 4255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:07.393981 4255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:07.394073 4255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:07.398355 4255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:07.398475 4255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:07.398511 4255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:07.398554 4255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:07.398590 4255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:07.404289 4255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:40263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:40263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:07.405534 4255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:07.406656 4255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:07.416242 4261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:07.416242 4263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:07.416290 4260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:07.416404 4255 server_base.cc:1034] running on GCE node
I20260926 13:40:07.417264 4255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:07.418565 4255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:07.419737 4255 hybrid_clock.cc:648] HybridClock initialized: now 1790430007419698 us; error 58 us; skew 500 ppm
Sep 26 13:40:07 dist-test-slave-1wns krb5kdc[4242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430007, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:07.422909 4255 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:40:07.423993 4255 webserver.cc:460] Webserver started at http://127.1.49.62:43147/ using document root <none> and password file <none>
I20260926 13:40:07.424544 4255 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:07.424603 4255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:07.424774 4255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:07.426908 4255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "0e67bccf287f4cf2925cd24b1e1cde3e"
format_stamp: "Formatted at 2026-09-26 13:40:07 on dist-test-slave-1wns"
server_key: "b179781b04f81c880746278f5278b590"
server_key_iv: "9ff61cdf20349532768ac1030276c6a9"
server_key_version: "encryptionkey@0"
I20260926 13:40:07.427474 4255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "0e67bccf287f4cf2925cd24b1e1cde3e"
format_stamp: "Formatted at 2026-09-26 13:40:07 on dist-test-slave-1wns"
server_key: "b179781b04f81c880746278f5278b590"
server_key_iv: "9ff61cdf20349532768ac1030276c6a9"
server_key_version: "encryptionkey@0"
I20260926 13:40:07.431612 4255 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 13:40:07.434553 4270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:07.436137 4255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:07.436305 4255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "0e67bccf287f4cf2925cd24b1e1cde3e"
format_stamp: "Formatted at 2026-09-26 13:40:07 on dist-test-slave-1wns"
server_key: "b179781b04f81c880746278f5278b590"
server_key_iv: "9ff61cdf20349532768ac1030276c6a9"
server_key_version: "encryptionkey@0"
I20260926 13:40:07.436440 4255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:07.457962 4255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:07.461501 4255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:07.461737 4255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:07.469630 4255 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:40263
I20260926 13:40:07.469947 4322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:40263 every 8 connection(s)
I20260926 13:40:07.470714 4255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:07.473553 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 4255
I20260926 13:40:07.473721 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:07.473775 4323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:07.474053 1220 external_mini_cluster.cc:1428] Setting key 9b5352312ed236a22d6c0da578529fba
I20260926 13:40:07.481215 4323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e: Bootstrap starting.
I20260926 13:40:07.484246 4323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e: Neither blocks nor log segments found. Creating new log.
Sep 26 13:40:07 dist-test-slave-1wns krb5kdc[4242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:07.485134 4323 log.cc:824] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:07.487717 4323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e: No bootstrap required, opened a new log
I20260926 13:40:07.490484 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:07.475690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54758 (local address 127.1.49.62:40263)
0926 13:40:07.476212 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:07.476233 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:07.476278 (+ 45us) server_negotiation.cc:407] Connection header received
0926 13:40:07.477227 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:07.477253 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:07.477713 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:07.478119 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:07.478884 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:07.480411 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:07.481481 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:07.482094 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:07.482294 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:07.485099 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:07.485106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:07.485123 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:07.485156 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:07.487295 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:07.487881 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:07.487889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:07.487899 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:07.488053 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:07.488361 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:07.488365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:07.488367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:07.488810 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:07.489019 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:07.489340 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 13:40:07.489687 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":191,"threads_started":1}
I20260926 13:40:07.491706 4323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e67bccf287f4cf2925cd24b1e1cde3e" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40263 } }
I20260926 13:40:07.492035 4323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:07.492118 4323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e67bccf287f4cf2925cd24b1e1cde3e, State: Initialized, Role: FOLLOWER
I20260926 13:40:07.492581 4323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e67bccf287f4cf2925cd24b1e1cde3e" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40263 } }
I20260926 13:40:07.492729 4323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:07.492802 4323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:07.492920 4323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:07.493934 4323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e67bccf287f4cf2925cd24b1e1cde3e" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40263 } }
I20260926 13:40:07.494254 4323 leader_election.cc:302] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e67bccf287f4cf2925cd24b1e1cde3e; no voters:
I20260926 13:40:07.494583 4323 leader_election.cc:288] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:07.494721 4328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:07.494964 4328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [term 1 LEADER]: Becoming Leader. State: Replica: 0e67bccf287f4cf2925cd24b1e1cde3e, State: Running, Role: LEADER
I20260926 13:40:07.495330 4328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e67bccf287f4cf2925cd24b1e1cde3e" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40263 } }
I20260926 13:40:07.495901 4323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:07.497099 4329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e67bccf287f4cf2925cd24b1e1cde3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e67bccf287f4cf2925cd24b1e1cde3e" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40263 } } }
I20260926 13:40:07.497246 4329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:07.497576 4330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e67bccf287f4cf2925cd24b1e1cde3e. Latest consensus state: current_term: 1 leader_uuid: "0e67bccf287f4cf2925cd24b1e1cde3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e67bccf287f4cf2925cd24b1e1cde3e" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40263 } } }
I20260926 13:40:07.497696 4330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:07.497720 4337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:07.501039 4337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:07.506559 4337 catalog_manager.cc:1347] Generated new cluster ID: 126a2b2fb61f4272b869c39a08d28e6c
I20260926 13:40:07.506649 4337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:07.529239 4337 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:07.530614 4337 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:07.543170 4337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e67bccf287f4cf2925cd24b1e1cde3e: Generated new TSK 0
I20260926 13:40:07.544096 4337 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:40:07.584808 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40263
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:07.704919 4349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:07.705278 4349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:07.705360 4349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:07.709857 4349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:07.709970 4349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:07.710074 4349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:40:07.716143 4349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:07.717379 4349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:07.718549 4349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:07.729099 4357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:07.729099 4354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:07.729151 4355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:07.729562 4349 server_base.cc:1034] running on GCE node
I20260926 13:40:07.730070 4349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:07.730935 4349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:07.732249 4349 hybrid_clock.cc:648] HybridClock initialized: now 1790430007732218 us; error 43 us; skew 500 ppm
Sep 26 13:40:07 dist-test-slave-1wns krb5kdc[4242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430007, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:07.735311 4349 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:40:07.736723 4349 webserver.cc:460] Webserver started at http://127.1.49.1:43877/ using document root <none> and password file <none>
I20260926 13:40:07.737349 4349 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:07.737457 4349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:07.737704 4349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:07.739987 4349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "510a193e7cc34d8cbae59a7317d7cfd2"
format_stamp: "Formatted at 2026-09-26 13:40:07 on dist-test-slave-1wns"
server_key: "a86fcef6c59a0a274c752c7c828ec81d"
server_key_iv: "704cfa09df8c8ee9527ea889b2ab1700"
server_key_version: "encryptionkey@0"
I20260926 13:40:07.740675 4349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "510a193e7cc34d8cbae59a7317d7cfd2"
format_stamp: "Formatted at 2026-09-26 13:40:07 on dist-test-slave-1wns"
server_key: "a86fcef6c59a0a274c752c7c828ec81d"
server_key_iv: "704cfa09df8c8ee9527ea889b2ab1700"
server_key_version: "encryptionkey@0"
I20260926 13:40:07.745215 4349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 13:40:07.748342 4364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:07.749788 4349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 13:40:07.749986 4349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "510a193e7cc34d8cbae59a7317d7cfd2"
format_stamp: "Formatted at 2026-09-26 13:40:07 on dist-test-slave-1wns"
server_key: "a86fcef6c59a0a274c752c7c828ec81d"
server_key_iv: "704cfa09df8c8ee9527ea889b2ab1700"
server_key_version: "encryptionkey@0"
I20260926 13:40:07.750133 4349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:07.769099 4349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:07.773252 4349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:07.773511 4349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:07.774205 4349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:07.775142 4349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:07.775219 4349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:07.775293 4349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:07.775362 4349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:07.785965 4349 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:42703
I20260926 13:40:07.786020 4477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:42703 every 8 connection(s)
I20260926 13:40:07.787022 4349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:40:07.793043 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 4349
I20260926 13:40:07.793200 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:40:07.793558 1220 external_mini_cluster.cc:1428] Setting key 8245e4dcefb0200d665f0656a8a4e237
Sep 26 13:40:07 dist-test-slave-1wns krb5kdc[4242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430007, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
I20260926 13:40:07.805706 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:07.789324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:53299 (local address 127.1.49.62:40263)
0926 13:40:07.789551 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:07.789557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:07.790628 (+ 1071us) server_negotiation.cc:407] Connection header received
0926 13:40:07.792309 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:07.792312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:07.792374 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:07.792531 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:07.794892 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:07.795423 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:07.797618 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:07.798054 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:07.798175 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:07.801269 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:07.801272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:07.801275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:07.801303 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:07.802992 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:07.803704 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:07.803710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:07.803717 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:07.803853 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:07.804280 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:07.804283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:07.804285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:07.804607 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:07.804733 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:07.805355 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 13:40:07.805498 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260926 13:40:07.806523 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:07.789894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:40263 (local address 127.1.49.1:53299)
0926 13:40:07.790412 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:07.790503 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:07.792043 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:07.792854 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:07.792866 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:07.793527 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:07.794266 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:07.794288 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:07.795586 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:07.795590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:07.797471 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:07.797483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:07.798245 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:07.798248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:07.798501 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:07.799268 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:07.799288 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:07.801042 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:07.803159 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:07.803166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:07.803186 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:07.803569 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:07.803981 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:07.803984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:07.803986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:07.804164 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:07.804743 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:07.804749 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:07.805067 (+ 318us) client_negotiation.cc:769] Sending connection context
0926 13:40:07.805248 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 13:40:07.805544 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":142,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
I20260926 13:40:07.808082 4478 heartbeater.cc:347] Connected to a master server at 127.1.49.62:40263
I20260926 13:40:07.808393 4478 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:07.809158 4478 heartbeater.cc:511] Master 127.1.49.62:40263 requested a full tablet report, sending...
I20260926 13:40:07.811184 4287 ts_manager.cc:194] Registered new tserver with Master: 510a193e7cc34d8cbae59a7317d7cfd2 (127.1.49.1:42703)
I20260926 13:40:07.812507 4287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:53299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:40:07.828109 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40263
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:07.945797 4483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:07.946113 4483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:07.946161 4483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:07.950459 4483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:07.950543 4483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:07.950639 4483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:40:07.956629 4483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:07.957875 4483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:07.958997 4483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:07.969372 4491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:07.969527 4483 server_base.cc:1034] running on GCE node
W20260926 13:40:07.969358 4488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:07.969746 4489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:07.970217 4483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:07.970902 4483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:07.972081 4483 hybrid_clock.cc:648] HybridClock initialized: now 1790430007972063 us; error 37 us; skew 500 ppm
Sep 26 13:40:07 dist-test-slave-1wns krb5kdc[4242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430007, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:07.975332 4483 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:40:07.976398 4483 webserver.cc:460] Webserver started at http://127.1.49.2:44019/ using document root <none> and password file <none>
I20260926 13:40:07.977021 4483 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:07.977087 4483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:07.977317 4483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:07.979568 4483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "e0fe3cb37d524ab391c5f41bc9983767"
format_stamp: "Formatted at 2026-09-26 13:40:07 on dist-test-slave-1wns"
server_key: "c5474ab10068acecec7f2a3a9f1973ba"
server_key_iv: "a2e836ed3abe3fa26cfb697860b35835"
server_key_version: "encryptionkey@0"
I20260926 13:40:07.980253 4483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "e0fe3cb37d524ab391c5f41bc9983767"
format_stamp: "Formatted at 2026-09-26 13:40:07 on dist-test-slave-1wns"
server_key: "c5474ab10068acecec7f2a3a9f1973ba"
server_key_iv: "a2e836ed3abe3fa26cfb697860b35835"
server_key_version: "encryptionkey@0"
I20260926 13:40:07.984530 4483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:40:07.987309 4498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:07.988864 4483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:07.989007 4483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "e0fe3cb37d524ab391c5f41bc9983767"
format_stamp: "Formatted at 2026-09-26 13:40:07 on dist-test-slave-1wns"
server_key: "c5474ab10068acecec7f2a3a9f1973ba"
server_key_iv: "a2e836ed3abe3fa26cfb697860b35835"
server_key_version: "encryptionkey@0"
I20260926 13:40:07.989140 4483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:08.004773 4483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:08.008661 4483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:08.008894 4483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:08.009559 4483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:08.010491 4483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:08.010565 4483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:08.010634 4483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:08.010679 4483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:08.021258 4483 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:35379
I20260926 13:40:08.021278 4611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:35379 every 8 connection(s)
I20260926 13:40:08.022627 4483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:40:08.026810 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 4483
I20260926 13:40:08.026947 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:40:08.027334 1220 external_mini_cluster.cc:1428] Setting key ef6d609b2a4286c6c6550010b5335990
Sep 26 13:40:08 dist-test-slave-1wns krb5kdc[4242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430007, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
I20260926 13:40:08.039592 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:08.024873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:52581 (local address 127.1.49.62:40263)
0926 13:40:08.025016 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:08.025020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:08.026818 (+ 1798us) server_negotiation.cc:407] Connection header received
0926 13:40:08.028713 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:08.028716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:08.028773 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:08.028879 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:08.030250 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:08.030788 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:08.032090 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:08.032309 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:08.032422 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:08.035455 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:08.035459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:08.035462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:08.035494 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:08.037154 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:08.037765 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:08.037769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:08.037773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:08.037896 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:08.038287 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:08.038291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:08.038293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:08.038505 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:08.038619 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:08.039271 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 13:40:08.039414 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
I20260926 13:40:08.040763 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:08.025501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:40263 (local address 127.1.49.2:52581)
0926 13:40:08.026545 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:08.026646 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:08.028489 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:08.029070 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:08.029078 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:08.029575 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:08.030091 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:08.030104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:08.030917 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:08.030920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:08.031952 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:08.031961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:08.032475 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:08.032479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:08.032658 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:08.033302 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:08.033319 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:08.035240 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:08.037301 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:08.037308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:08.037329 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:08.037642 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:08.038011 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:08.038014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:08.038016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:08.038186 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:08.038616 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:08.038624 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:08.038994 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 13:40:08.039226 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 13:40:08.039622 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":203,"threads_started":1}
I20260926 13:40:08.042076 4612 heartbeater.cc:347] Connected to a master server at 127.1.49.62:40263
I20260926 13:40:08.042385 4612 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:08.043081 4612 heartbeater.cc:511] Master 127.1.49.62:40263 requested a full tablet report, sending...
I20260926 13:40:08.044324 4287 ts_manager.cc:194] Registered new tserver with Master: e0fe3cb37d524ab391c5f41bc9983767 (127.1.49.2:35379)
I20260926 13:40:08.044942 4287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:52581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:40:08.060282 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40263
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:08.182741 4617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:08.183050 4617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:08.183099 4617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:08.187485 4617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:08.187566 4617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:08.187630 4617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:40:08.193360 4617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:08.194621 4617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:08.195786 4617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:08.206707 4623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:08.206833 4617 server_base.cc:1034] running on GCE node
W20260926 13:40:08.206703 4625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:08.206703 4622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:08.207470 4617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:08.208427 4617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:08.209647 4617 hybrid_clock.cc:648] HybridClock initialized: now 1790430008209610 us; error 55 us; skew 500 ppm
Sep 26 13:40:08 dist-test-slave-1wns krb5kdc[4242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430008, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:08.213011 4617 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:40:08.214233 4617 webserver.cc:460] Webserver started at http://127.1.49.3:36527/ using document root <none> and password file <none>
I20260926 13:40:08.214938 4617 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:08.215052 4617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:08.215322 4617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:08.217856 4617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "8e5a1c0e97f04c29ab75457dbcfc8add"
format_stamp: "Formatted at 2026-09-26 13:40:08 on dist-test-slave-1wns"
server_key: "ea216233e0a87721da90ef2b3935e0ff"
server_key_iv: "e8140e95f4b56ca13b86d48c0aced71f"
server_key_version: "encryptionkey@0"
I20260926 13:40:08.218828 4617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "8e5a1c0e97f04c29ab75457dbcfc8add"
format_stamp: "Formatted at 2026-09-26 13:40:08 on dist-test-slave-1wns"
server_key: "ea216233e0a87721da90ef2b3935e0ff"
server_key_iv: "e8140e95f4b56ca13b86d48c0aced71f"
server_key_version: "encryptionkey@0"
I20260926 13:40:08.223645 4617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 13:40:08.226953 4632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:08.228732 4617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:08.228910 4617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "8e5a1c0e97f04c29ab75457dbcfc8add"
format_stamp: "Formatted at 2026-09-26 13:40:08 on dist-test-slave-1wns"
server_key: "ea216233e0a87721da90ef2b3935e0ff"
server_key_iv: "e8140e95f4b56ca13b86d48c0aced71f"
server_key_version: "encryptionkey@0"
I20260926 13:40:08.229063 4617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:08.255723 4617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:08.259732 4617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:08.260005 4617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:08.260671 4617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:08.261560 4617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:08.261636 4617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:08.261703 4617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:08.261758 4617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:08.271725 4617 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:45333
I20260926 13:40:08.271822 4745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:45333 every 8 connection(s)
I20260926 13:40:08.272781 4617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:40:08.278376 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 4617
I20260926 13:40:08.278571 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:40:08.278880 1220 external_mini_cluster.cc:1428] Setting key c00b4819ca825d0bf0bac501131fcad5
Sep 26 13:40:08 dist-test-slave-1wns krb5kdc[4242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430008, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:08.290215 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:08.274919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:41705 (local address 127.1.49.62:40263)
0926 13:40:08.275091 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:08.275095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:08.276157 (+ 1062us) server_negotiation.cc:407] Connection header received
0926 13:40:08.277681 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:08.277685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:08.277747 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:08.277856 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:08.279508 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:08.280414 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:08.282007 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:08.282252 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:08.282357 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:08.285562 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:08.285566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:08.285568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:08.285600 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:08.287548 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:08.288258 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:08.288261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:08.288263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:08.288430 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:08.288980 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:08.288983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:08.288984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:08.289207 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:08.289292 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:08.289959 (+ 667us) server_negotiation.cc:299] Negotiation successful
0926 13:40:08.290081 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 13:40:08.291128 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:08.275364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:40263 (local address 127.1.49.3:41705)
0926 13:40:08.275982 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:08.276031 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:08.277370 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:08.278048 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:08.278056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:08.278501 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:08.279189 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:08.279220 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:08.280624 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:08.280627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:08.281819 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:08.281830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:08.282415 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:08.282418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:08.282641 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:08.283286 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:08.283303 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:08.285322 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:08.287687 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:08.287693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:08.287709 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:08.288147 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:08.288662 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:08.288664 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:08.288666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:08.288839 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:08.289327 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:08.289333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:08.289695 (+ 362us) client_negotiation.cc:769] Sending connection context
0926 13:40:08.289877 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 13:40:08.290183 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":146,"threads_started":1}
I20260926 13:40:08.292729 4746 heartbeater.cc:347] Connected to a master server at 127.1.49.62:40263
I20260926 13:40:08.293004 4746 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:08.293648 4746 heartbeater.cc:511] Master 127.1.49.62:40263 requested a full tablet report, sending...
I20260926 13:40:08.294798 4287 ts_manager.cc:194] Registered new tserver with Master: 8e5a1c0e97f04c29ab75457dbcfc8add (127.1.49.3:45333)
I20260926 13:40:08.295506 4287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:41705
I20260926 13:40:08.306147 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:40:08 dist-test-slave-1wns krb5kdc[4242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430008, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 13:40:08 dist-test-slave-1wns krb5kdc[4242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430008, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:08.335960 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:08.325988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54764 (local address 127.1.49.62:40263)
0926 13:40:08.326140 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:08.326144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:08.326313 (+ 169us) server_negotiation.cc:407] Connection header received
0926 13:40:08.326427 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:08.326430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:08.326482 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:08.326582 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:08.327370 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:08.328092 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:08.328976 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:08.329212 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:08.329303 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:08.332014 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:08.332017 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:08.332019 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:08.332048 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:08.333816 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:08.334364 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:08.334368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:08.334369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:08.334479 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:08.334875 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:08.334879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:08.334880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:08.335154 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:08.335280 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:08.335623 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 13:40:08.335733 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 13:40:08.336431 4287 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54764
I20260926 13:40:08.337703 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 4349
I20260926 13:40:08.344205 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 4483
I20260926 13:40:08.349658 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 4617
I20260926 13:40:08.355389 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 4255
2026-09-26T13:40:08Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2294 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:40:08 dist-test-slave-1wns krb5kdc[4758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:40:08 dist-test-slave-1wns krb5kdc[4758](info): set up 2 sockets
Sep 26 13:40:08 dist-test-slave-1wns krb5kdc[4758](info): commencing operation
krb5kdc: starting...
W20260926 13:40:09.416939 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:40:09 dist-test-slave-1wns krb5kdc[4758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:40:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:40:09.564733 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:46813
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:46813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:09.686380 4771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:09.686779 4771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:09.686859 4771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:09.691227 4771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:09.691350 4771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:09.691385 4771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:09.691424 4771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:09.691483 4771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:09.697583 4771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:46813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:46813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:09.698896 4771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:09.700102 4771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:09.709928 4777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:09.709934 4776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:09.709949 4779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:09.710325 4771 server_base.cc:1034] running on GCE node
I20260926 13:40:09.711192 4771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:09.712551 4771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:09.713776 4771 hybrid_clock.cc:648] HybridClock initialized: now 1790430009713755 us; error 38 us; skew 500 ppm
Sep 26 13:40:09 dist-test-slave-1wns krb5kdc[4758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430009, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:09.723289 4771 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:40:09.724664 4771 webserver.cc:460] Webserver started at http://127.1.49.62:42383/ using document root <none> and password file <none>
I20260926 13:40:09.725425 4771 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:09.725544 4771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:09.725924 4771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:09.728533 4771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "ebc8e1e2c09c410cb73fdf9150ca47e2"
format_stamp: "Formatted at 2026-09-26 13:40:09 on dist-test-slave-1wns"
server_key: "e58182389f6b3ba2d5510088975585f4"
server_key_iv: "4dd7ddaa3ecc4fbc46b6be49b93dae9c"
server_key_version: "encryptionkey@0"
I20260926 13:40:09.729252 4771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "ebc8e1e2c09c410cb73fdf9150ca47e2"
format_stamp: "Formatted at 2026-09-26 13:40:09 on dist-test-slave-1wns"
server_key: "e58182389f6b3ba2d5510088975585f4"
server_key_iv: "4dd7ddaa3ecc4fbc46b6be49b93dae9c"
server_key_version: "encryptionkey@0"
I20260926 13:40:09.733968 4771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 13:40:09.737227 4786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:09.738844 4771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:40:09.739048 4771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "ebc8e1e2c09c410cb73fdf9150ca47e2"
format_stamp: "Formatted at 2026-09-26 13:40:09 on dist-test-slave-1wns"
server_key: "e58182389f6b3ba2d5510088975585f4"
server_key_iv: "4dd7ddaa3ecc4fbc46b6be49b93dae9c"
server_key_version: "encryptionkey@0"
I20260926 13:40:09.739220 4771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:09.758962 4771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:09.762940 4771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:09.763181 4771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:09.771539 4771 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:46813
I20260926 13:40:09.771710 4838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:46813 every 8 connection(s)
I20260926 13:40:09.772723 4771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:09.773034 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 4771
I20260926 13:40:09.773175 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:09.773514 1220 external_mini_cluster.cc:1428] Setting key cfaba812b5411188ff7b2aa2bd7fafde
I20260926 13:40:09.776551 4839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:09.783129 4839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2: Bootstrap starting.
Sep 26 13:40:09 dist-test-slave-1wns krb5kdc[4758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:09.785842 4839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:09.786665 4839 log.cc:824] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:09.789278 4839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2: No bootstrap required, opened a new log
I20260926 13:40:09.790346 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:09.775192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48706 (local address 127.1.49.62:46813)
0926 13:40:09.775700 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:09.775716 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:09.775756 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:40:09.776574 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:09.776592 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:09.777045 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:09.777424 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:09.778415 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:09.779490 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:09.780465 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:09.780989 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:09.781186 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:09.784319 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:09.784326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:09.784343 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:09.784382 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:09.786583 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:09.787205 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:09.787212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:09.787217 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:09.787353 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:09.787709 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:09.787713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:09.787715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:09.788281 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:09.788509 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:09.788895 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 13:40:09.789234 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":172,"threads_started":1}
I20260926 13:40:09.793756 4839 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc8e1e2c09c410cb73fdf9150ca47e2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46813 } }
I20260926 13:40:09.794003 4839 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:09.794047 4839 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc8e1e2c09c410cb73fdf9150ca47e2, State: Initialized, Role: FOLLOWER
I20260926 13:40:09.794420 4839 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc8e1e2c09c410cb73fdf9150ca47e2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46813 } }
I20260926 13:40:09.794529 4839 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:09.794576 4839 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:09.794652 4839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:09.795657 4839 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc8e1e2c09c410cb73fdf9150ca47e2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46813 } }
I20260926 13:40:09.796103 4839 leader_election.cc:302] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebc8e1e2c09c410cb73fdf9150ca47e2; no voters:
I20260926 13:40:09.796455 4839 leader_election.cc:288] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:09.796703 4844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:09.797067 4844 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [term 1 LEADER]: Becoming Leader. State: Replica: ebc8e1e2c09c410cb73fdf9150ca47e2, State: Running, Role: LEADER
I20260926 13:40:09.797441 4844 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc8e1e2c09c410cb73fdf9150ca47e2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46813 } }
I20260926 13:40:09.797907 4839 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:09.799463 4845 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebc8e1e2c09c410cb73fdf9150ca47e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc8e1e2c09c410cb73fdf9150ca47e2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46813 } } }
I20260926 13:40:09.799641 4845 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:09.800084 4846 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebc8e1e2c09c410cb73fdf9150ca47e2. Latest consensus state: current_term: 1 leader_uuid: "ebc8e1e2c09c410cb73fdf9150ca47e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc8e1e2c09c410cb73fdf9150ca47e2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46813 } } }
I20260926 13:40:09.800156 4857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:09.800205 4846 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:09.804304 4857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:09.811349 4857 catalog_manager.cc:1347] Generated new cluster ID: 4f84a80ef44349bf939c9917902cf188
I20260926 13:40:09.811457 4857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:09.871084 4857 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:09.872534 4857 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:09.880046 4857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebc8e1e2c09c410cb73fdf9150ca47e2: Generated new TSK 0
I20260926 13:40:09.881048 4857 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:40:09.924712 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:46813
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:10.049957 4865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:10.050294 4865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:10.050343 4865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:10.054818 4865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:10.054911 4865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:10.054981 4865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:40:10.060798 4865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:46813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:10.062081 4865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:10.063270 4865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:10.074352 4873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:10.074352 4871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:10.074457 4870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:10.074754 4865 server_base.cc:1034] running on GCE node
I20260926 13:40:10.075686 4865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:10.076509 4865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:10.077700 4865 hybrid_clock.cc:648] HybridClock initialized: now 1790430010077680 us; error 40 us; skew 500 ppm
Sep 26 13:40:10 dist-test-slave-1wns krb5kdc[4758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:10.081094 4865 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:40:10.082162 4865 webserver.cc:460] Webserver started at http://127.1.49.1:38689/ using document root <none> and password file <none>
I20260926 13:40:10.082788 4865 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:10.082852 4865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:10.083043 4865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:10.085367 4865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "50d8e9ebe0aa4071846008e1fe2d6f15"
format_stamp: "Formatted at 2026-09-26 13:40:10 on dist-test-slave-1wns"
server_key: "3dce0a21b9f3c9a08ee8f98dd3f48ed3"
server_key_iv: "2b7e2b1c6a0daa4a28e6f47723e9ec75"
server_key_version: "encryptionkey@0"
I20260926 13:40:10.085950 4865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "50d8e9ebe0aa4071846008e1fe2d6f15"
format_stamp: "Formatted at 2026-09-26 13:40:10 on dist-test-slave-1wns"
server_key: "3dce0a21b9f3c9a08ee8f98dd3f48ed3"
server_key_iv: "2b7e2b1c6a0daa4a28e6f47723e9ec75"
server_key_version: "encryptionkey@0"
I20260926 13:40:10.090337 4865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:40:10.093315 4880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:10.095005 4865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:40:10.095193 4865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "50d8e9ebe0aa4071846008e1fe2d6f15"
format_stamp: "Formatted at 2026-09-26 13:40:10 on dist-test-slave-1wns"
server_key: "3dce0a21b9f3c9a08ee8f98dd3f48ed3"
server_key_iv: "2b7e2b1c6a0daa4a28e6f47723e9ec75"
server_key_version: "encryptionkey@0"
I20260926 13:40:10.095335 4865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:10.111688 4865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:10.115737 4865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:10.116034 4865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:10.116868 4865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:10.117890 4865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:10.117949 4865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:10.118021 4865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:10.118057 4865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:10.128962 4865 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:32845
I20260926 13:40:10.129001 4993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:32845 every 8 connection(s)
I20260926 13:40:10.130043 4865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:40:10.133622 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 4865
I20260926 13:40:10.133770 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:40:10.134075 1220 external_mini_cluster.cc:1428] Setting key 17e4200b93d9e38aa4c2d3a7f9dea4f9
Sep 26 13:40:10 dist-test-slave-1wns krb5kdc[4758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
I20260926 13:40:10.147485 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:10.132242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:38309 (local address 127.1.49.62:46813)
0926 13:40:10.132412 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:10.132416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:10.133349 (+ 933us) server_negotiation.cc:407] Connection header received
0926 13:40:10.135234 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:10.135238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:10.135294 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:10.135374 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:10.136816 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.137378 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.138979 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.139229 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.139324 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:10.142772 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:10.142778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:10.142783 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:10.142819 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:10.145091 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:10.145714 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:10.145718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:10.145720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:10.145820 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:10.146230 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:10.146234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:10.146236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:10.146453 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:10.146581 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:10.147165 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 13:40:10.147315 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
I20260926 13:40:10.148729 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:10.132601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46813 (local address 127.1.49.1:38309)
0926 13:40:10.133134 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:10.133186 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:10.134995 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:10.135575 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:10.135584 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:10.136118 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:10.136652 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:10.136667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.137594 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.137600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:10.138778 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:10.138795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.139432 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.139439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:10.139718 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:10.140574 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:10.140591 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:10.142521 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:10.145252 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:10.145259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:10.145289 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:10.145593 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:10.145947 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:10.145950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:10.145952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:10.146124 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:10.146602 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:10.146609 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:10.146893 (+ 284us) client_negotiation.cc:769] Sending connection context
0926 13:40:10.147162 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 13:40:10.147618 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":126,"threads_started":1}
I20260926 13:40:10.150169 4994 heartbeater.cc:347] Connected to a master server at 127.1.49.62:46813
I20260926 13:40:10.150462 4994 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:10.151013 4994 heartbeater.cc:511] Master 127.1.49.62:46813 requested a full tablet report, sending...
I20260926 13:40:10.152740 4801 ts_manager.cc:194] Registered new tserver with Master: 50d8e9ebe0aa4071846008e1fe2d6f15 (127.1.49.1:32845)
I20260926 13:40:10.154055 4801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:38309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:40:10.168133 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:46813
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:10.291332 4999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:10.291678 4999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:10.291754 4999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:10.296205 4999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:10.296303 4999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:10.296399 4999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:40:10.302762 4999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:46813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:10.304086 4999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:10.305178 4999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:10.316254 5007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:10.316282 5004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:10.316311 4999 server_base.cc:1034] running on GCE node
W20260926 13:40:10.316665 5005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:10.317159 4999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:10.318101 4999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:10.319348 4999 hybrid_clock.cc:648] HybridClock initialized: now 1790430010319304 us; error 66 us; skew 500 ppm
Sep 26 13:40:10 dist-test-slave-1wns krb5kdc[4758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:10.322512 4999 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:40:10.323593 4999 webserver.cc:460] Webserver started at http://127.1.49.2:42817/ using document root <none> and password file <none>
I20260926 13:40:10.324249 4999 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:10.324338 4999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:10.324587 4999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:10.326776 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "c967b08de1014af898477fdc88a2122c"
format_stamp: "Formatted at 2026-09-26 13:40:10 on dist-test-slave-1wns"
server_key: "a42661f3bcb4569928594862af4727d1"
server_key_iv: "a93633de504b88fc82334b2c07054444"
server_key_version: "encryptionkey@0"
I20260926 13:40:10.327422 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "c967b08de1014af898477fdc88a2122c"
format_stamp: "Formatted at 2026-09-26 13:40:10 on dist-test-slave-1wns"
server_key: "a42661f3bcb4569928594862af4727d1"
server_key_iv: "a93633de504b88fc82334b2c07054444"
server_key_version: "encryptionkey@0"
I20260926 13:40:10.331565 4999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 13:40:10.334468 5014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:10.335942 4999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:10.336089 4999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "c967b08de1014af898477fdc88a2122c"
format_stamp: "Formatted at 2026-09-26 13:40:10 on dist-test-slave-1wns"
server_key: "a42661f3bcb4569928594862af4727d1"
server_key_iv: "a93633de504b88fc82334b2c07054444"
server_key_version: "encryptionkey@0"
I20260926 13:40:10.336239 4999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:10.368682 4999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:10.373117 4999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:10.373389 4999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:10.374079 4999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:10.375056 4999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:10.375139 4999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:10.375214 4999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:10.375258 4999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:10.385680 4999 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:33827
I20260926 13:40:10.385703 5127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:33827 every 8 connection(s)
I20260926 13:40:10.386754 4999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:40:10.396339 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 4999
I20260926 13:40:10.396492 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:40:10.396771 1220 external_mini_cluster.cc:1428] Setting key 8e0c4bd9969e7cb302736248856d0dfb
Sep 26 13:40:10 dist-test-slave-1wns krb5kdc[4758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:10.403427 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:10.389096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:48733 (local address 127.1.49.62:46813)
0926 13:40:10.389257 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:10.389261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:10.390315 (+ 1054us) server_negotiation.cc:407] Connection header received
0926 13:40:10.391860 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:10.391867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:10.391933 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:10.392035 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:10.393403 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.394022 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.395273 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.395533 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.395628 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:10.398845 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:10.398852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:10.398857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:10.398890 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:10.400898 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:10.401558 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:10.401564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:10.401566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:10.401655 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:10.402081 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:10.402085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:10.402086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:10.402291 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:10.402380 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:10.403093 (+ 713us) server_negotiation.cc:299] Negotiation successful
0926 13:40:10.403257 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 13:40:10.404814 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:10.389572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46813 (local address 127.1.49.2:48733)
0926 13:40:10.390136 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:10.390186 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:10.391531 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:10.392232 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:10.392239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:10.392638 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:10.393231 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:10.393247 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.394151 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.394153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:10.395135 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:10.395144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.395713 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.395716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:10.395957 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:10.396682 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:10.396699 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:10.398615 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:10.401121 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:10.401127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:10.401144 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:10.401442 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:10.401757 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:10.401764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:10.401769 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:10.401972 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:10.402415 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:10.402425 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:10.402798 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 13:40:10.403043 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 13:40:10.403479 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":143,"threads_started":1}
I20260926 13:40:10.406177 5128 heartbeater.cc:347] Connected to a master server at 127.1.49.62:46813
I20260926 13:40:10.406486 5128 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:10.407176 5128 heartbeater.cc:511] Master 127.1.49.62:46813 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
I20260926 13:40:10.408599 4801 ts_manager.cc:194] Registered new tserver with Master: c967b08de1014af898477fdc88a2122c (127.1.49.2:33827)
I20260926 13:40:10.409250 4801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:48733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:40:10.431583 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:46813
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:10.554414 5133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:10.554741 5133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:10.554792 5133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:10.559211 5133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:10.559312 5133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:10.559386 5133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:40:10.565022 5133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:46813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:10.566238 5133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:10.567592 5133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:10.578765 5139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:10.578751 5138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:10.578842 5133 server_base.cc:1034] running on GCE node
W20260926 13:40:10.578765 5141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:10.579548 5133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:10.580349 5133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:10.581580 5133 hybrid_clock.cc:648] HybridClock initialized: now 1790430010581557 us; error 52 us; skew 500 ppm
Sep 26 13:40:10 dist-test-slave-1wns krb5kdc[4758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:10.584965 5133 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:40:10.586167 5133 webserver.cc:460] Webserver started at http://127.1.49.3:39845/ using document root <none> and password file <none>
I20260926 13:40:10.586812 5133 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:10.586915 5133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:10.587182 5133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:10.589547 5133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "312e89d99e9d4a269242f773f8067793"
format_stamp: "Formatted at 2026-09-26 13:40:10 on dist-test-slave-1wns"
server_key: "c2f375ae00141b54f5cc3af9c0f7c47a"
server_key_iv: "65a9e82b3a4aca086d3632693efdbc44"
server_key_version: "encryptionkey@0"
I20260926 13:40:10.590279 5133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "312e89d99e9d4a269242f773f8067793"
format_stamp: "Formatted at 2026-09-26 13:40:10 on dist-test-slave-1wns"
server_key: "c2f375ae00141b54f5cc3af9c0f7c47a"
server_key_iv: "65a9e82b3a4aca086d3632693efdbc44"
server_key_version: "encryptionkey@0"
I20260926 13:40:10.594648 5133 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 13:40:10.597730 5148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:10.599785 5133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:40:10.600055 5133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "312e89d99e9d4a269242f773f8067793"
format_stamp: "Formatted at 2026-09-26 13:40:10 on dist-test-slave-1wns"
server_key: "c2f375ae00141b54f5cc3af9c0f7c47a"
server_key_iv: "65a9e82b3a4aca086d3632693efdbc44"
server_key_version: "encryptionkey@0"
I20260926 13:40:10.600232 5133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:10.619165 5133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:10.623128 5133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:10.623308 5133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:10.623996 5133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:10.624858 5133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:10.624914 5133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:10.624956 5133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:10.624971 5133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:10.634864 5133 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:32879
I20260926 13:40:10.634881 5261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:32879 every 8 connection(s)
I20260926 13:40:10.635957 5133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:40:10.639704 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 5133
I20260926 13:40:10.639874 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:40:10.640163 1220 external_mini_cluster.cc:1428] Setting key e8d95f842a3e317edfe610d3eaddee50
Sep 26 13:40:10 dist-test-slave-1wns krb5kdc[4758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:10.652683 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:10.638255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:49907 (local address 127.1.49.62:46813)
0926 13:40:10.638447 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:10.638450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:10.639361 (+ 911us) server_negotiation.cc:407] Connection header received
0926 13:40:10.641200 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:10.641204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:10.641263 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:10.641344 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:10.642873 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.643600 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.645193 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.645419 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.645526 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:10.648467 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:10.648470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:10.648472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:10.648501 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:10.650219 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:10.650905 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:10.650909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:10.650911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:10.651014 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:10.651397 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:10.651408 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:10.651409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:10.651601 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:10.651715 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:10.652338 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 13:40:10.652511 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 13:40:10.653800 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:10.638667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46813 (local address 127.1.49.3:49907)
0926 13:40:10.639174 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:10.639226 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:10.640930 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:10.641557 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:10.641567 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:10.642123 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:10.642736 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:10.642752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.643733 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.643737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:10.644988 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:10.644999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.645639 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.645641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:10.645797 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:10.646432 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:10.646451 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:10.648246 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:10.650422 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:10.650430 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:10.650451 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:10.650775 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:10.651117 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:10.651121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:10.651123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:10.651279 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:10.651713 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:10.651721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:10.652079 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 13:40:10.652279 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 13:40:10.652671 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":129,"threads_started":1}
I20260926 13:40:10.655407 5262 heartbeater.cc:347] Connected to a master server at 127.1.49.62:46813
I20260926 13:40:10.655692 5262 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:10.656268 5262 heartbeater.cc:511] Master 127.1.49.62:46813 requested a full tablet report, sending...
I20260926 13:40:10.657483 4801 ts_manager.cc:194] Registered new tserver with Master: 312e89d99e9d4a269242f773f8067793 (127.1.49.3:32879)
I20260926 13:40:10.658116 4801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:49907
I20260926 13:40:10.667466 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:40:10 dist-test-slave-1wns krb5kdc[4758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:40:10 dist-test-slave-1wns krb5kdc[4758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.1@KRBTEST.COM
I20260926 13:40:10.698511 5269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:10.685933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.1.49.1:32845)
0926 13:40:10.686323 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:10.686330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:10.686357 (+ 27us) server_negotiation.cc:407] Connection header received
0926 13:40:10.686412 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:10.686418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:10.686713 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:10.686866 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:10.688016 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.688996 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.689996 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.690596 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.690709 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:10.693794 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:10.693809 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:10.693825 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:10.693859 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:10.696397 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:10.696971 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:10.696978 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:10.696979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:10.697098 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:10.697442 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:10.697444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:10.697445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:10.697714 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:10.697886 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:10.698155 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 13:40:10.698368 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":117,"threads_started":1}
Sep 26 13:40:10 dist-test-slave-1wns krb5kdc[4758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:10.713127 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:10.702790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48712 (local address 127.1.49.62:46813)
0926 13:40:10.702940 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:10.702943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:10.702957 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:40:10.703000 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:10.703002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:10.703043 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:10.703120 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:10.703961 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.704659 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.705636 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:10.705855 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:10.705942 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:10.708719 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:10.708723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:10.708725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:10.708756 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:10.710293 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:10.710945 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:10.710948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:10.710949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:10.711049 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:10.711642 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:10.711645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:10.711647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:10.712127 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:10.712311 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:10.712723 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 13:40:10.712845 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 13:40:10.713667 4801 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48712
I20260926 13:40:10.714556 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 4865
I20260926 13:40:10.721405 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 4999
I20260926 13:40:10.727066 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 5133
I20260926 13:40:10.733450 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 4771
2026-09-26T13:40:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2378 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:40:10 dist-test-slave-1wns krb5kdc[5274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:40:10 dist-test-slave-1wns krb5kdc[5274](info): set up 2 sockets
Sep 26 13:40:10 dist-test-slave-1wns krb5kdc[5274](info): commencing operation
krb5kdc: starting...
W20260926 13:40:11.850250 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:40:11 dist-test-slave-1wns krb5kdc[5274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:40:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:40:11.988633 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:43391
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:43391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:12.113937 5287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:12.114274 5287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:12.114322 5287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:12.118933 5287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:12.119033 5287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:12.119055 5287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:12.119073 5287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:12.119089 5287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:12.125134 5287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:43391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:43391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:12.126439 5287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:12.127924 5287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:40:12.137312 5287 server_base.cc:1034] running on GCE node
W20260926 13:40:12.137235 5292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:12.137216 5293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:12.137249 5295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:12.138358 5287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:12.139662 5287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:12.140848 5287 hybrid_clock.cc:648] HybridClock initialized: now 1790430012140803 us; error 74 us; skew 500 ppm
Sep 26 13:40:12 dist-test-slave-1wns krb5kdc[5274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430012, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:12.144749 5287 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:40:12.145669 5287 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:12.145823 5287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:12.146095 5287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:12.148658 5287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "ec0d73a629b340dfa1ecf87c673f5fb1"
format_stamp: "Formatted at 2026-09-26 13:40:12 on dist-test-slave-1wns"
server_key: "d2aac15bd6314de8ec7e9533df718044"
server_key_iv: "a2599866eb53bb6c03ada286515c47ef"
server_key_version: "encryptionkey@0"
I20260926 13:40:12.149457 5287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "ec0d73a629b340dfa1ecf87c673f5fb1"
format_stamp: "Formatted at 2026-09-26 13:40:12 on dist-test-slave-1wns"
server_key: "d2aac15bd6314de8ec7e9533df718044"
server_key_iv: "a2599866eb53bb6c03ada286515c47ef"
server_key_version: "encryptionkey@0"
I20260926 13:40:12.154233 5287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:40:12.157217 5301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:12.158840 5287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:40:12.159010 5287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "ec0d73a629b340dfa1ecf87c673f5fb1"
format_stamp: "Formatted at 2026-09-26 13:40:12 on dist-test-slave-1wns"
server_key: "d2aac15bd6314de8ec7e9533df718044"
server_key_iv: "a2599866eb53bb6c03ada286515c47ef"
server_key_version: "encryptionkey@0"
I20260926 13:40:12.159150 5287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:12.197050 5287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:12.201099 5287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:12.201352 5287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:12.210206 5287 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:43391
I20260926 13:40:12.210517 5353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:43391 every 8 connection(s)
I20260926 13:40:12.211069 5287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:12.214243 5354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:12.217609 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 5287
I20260926 13:40:12.217782 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:12.218075 1220 external_mini_cluster.cc:1428] Setting key f880eb71fc1b67c2c654bf19f55baa6e
I20260926 13:40:12.221544 5354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1: Bootstrap starting.
I20260926 13:40:12.224153 5354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:12.225044 5354 log.cc:824] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:12.227995 5354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1: No bootstrap required, opened a new log
Sep 26 13:40:12 dist-test-slave-1wns krb5kdc[5274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:12.232975 5354 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0d73a629b340dfa1ecf87c673f5fb1" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 43391 } }
I20260926 13:40:12.233302 5354 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:12.233403 5354 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0d73a629b340dfa1ecf87c673f5fb1, State: Initialized, Role: FOLLOWER
I20260926 13:40:12.233964 5354 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0d73a629b340dfa1ecf87c673f5fb1" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 43391 } }
I20260926 13:40:12.234164 5354 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:12.234273 5354 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:12.234403 5354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:12.235548 5354 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0d73a629b340dfa1ecf87c673f5fb1" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 43391 } }
I20260926 13:40:12.235877 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:12.219725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.1.49.62:43391)
0926 13:40:12.220283 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:12.220299 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:12.220337 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:40:12.221086 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:12.221105 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:12.221626 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:12.222023 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:12.222910 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:12.224021 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:12.225006 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:12.225654 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:12.225853 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:12.229175 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:12.229183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:12.229203 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:12.229242 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:12.232123 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:12.232698 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:12.232705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:12.232712 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:12.232886 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:12.233288 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:12.233292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:12.233294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:12.233913 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:12.234133 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:12.234584 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 13:40:12.234909 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":159,"threads_started":1}
I20260926 13:40:12.236064 5354 leader_election.cc:302] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec0d73a629b340dfa1ecf87c673f5fb1; no voters:
I20260926 13:40:12.236487 5354 leader_election.cc:288] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:12.236613 5359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:12.236972 5359 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [term 1 LEADER]: Becoming Leader. State: Replica: ec0d73a629b340dfa1ecf87c673f5fb1, State: Running, Role: LEADER
I20260926 13:40:12.237401 5359 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0d73a629b340dfa1ecf87c673f5fb1" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 43391 } }
I20260926 13:40:12.237900 5354 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:12.238828 5360 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec0d73a629b340dfa1ecf87c673f5fb1. Latest consensus state: current_term: 1 leader_uuid: "ec0d73a629b340dfa1ecf87c673f5fb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0d73a629b340dfa1ecf87c673f5fb1" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 43391 } } }
I20260926 13:40:12.238994 5360 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:12.239454 5368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:12.239871 5361 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec0d73a629b340dfa1ecf87c673f5fb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0d73a629b340dfa1ecf87c673f5fb1" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 43391 } } }
I20260926 13:40:12.240082 5361 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:12.243085 5368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:12.249504 5368 catalog_manager.cc:1347] Generated new cluster ID: d450406ed03f4c83960f3c0e98d81cca
I20260926 13:40:12.249635 5368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:12.259522 5368 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:12.260946 5368 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:12.273857 5368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1: Generated new TSK 0
I20260926 13:40:12.274890 5368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:40:12.311792 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:43391
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:12.450286 5380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:12.450601 5380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:12.450654 5380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:12.454896 5380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:12.454977 5380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:12.455040 5380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:40:12.461971 5380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:43391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:12.463307 5380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:12.464495 5380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:12.474900 5385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:12.474902 5386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:12.474900 5388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:12.475255 5380 server_base.cc:1034] running on GCE node
I20260926 13:40:12.475795 5380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:12.476614 5380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:12.477854 5380 hybrid_clock.cc:648] HybridClock initialized: now 1790430012477821 us; error 48 us; skew 500 ppm
Sep 26 13:40:12 dist-test-slave-1wns krb5kdc[5274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430012, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:12.481220 5380 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:40:12.482203 5380 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:12.482311 5380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:12.482571 5380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:12.484885 5380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "3861dfe019934bee8b2121a538493020"
format_stamp: "Formatted at 2026-09-26 13:40:12 on dist-test-slave-1wns"
server_key: "8475c43daf32b2dfdd1057260fcf0e7f"
server_key_iv: "2ea1d7094fcf8b40ee5a42e5f3ba8db1"
server_key_version: "encryptionkey@0"
I20260926 13:40:12.485519 5380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "3861dfe019934bee8b2121a538493020"
format_stamp: "Formatted at 2026-09-26 13:40:12 on dist-test-slave-1wns"
server_key: "8475c43daf32b2dfdd1057260fcf0e7f"
server_key_iv: "2ea1d7094fcf8b40ee5a42e5f3ba8db1"
server_key_version: "encryptionkey@0"
I20260926 13:40:12.489847 5380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:40:12.492906 5394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:12.494535 5380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:12.494724 5380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "3861dfe019934bee8b2121a538493020"
format_stamp: "Formatted at 2026-09-26 13:40:12 on dist-test-slave-1wns"
server_key: "8475c43daf32b2dfdd1057260fcf0e7f"
server_key_iv: "2ea1d7094fcf8b40ee5a42e5f3ba8db1"
server_key_version: "encryptionkey@0"
I20260926 13:40:12.494874 5380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:12.512938 5380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:12.516538 5380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:12.516773 5380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:12.517423 5380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:12.518318 5380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:12.518393 5380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:12.518461 5380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:12.518507 5380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:12.528738 5380 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:36553
I20260926 13:40:12.528764 5507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:36553 every 8 connection(s)
I20260926 13:40:12.529484 5380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:40:12.539572 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 5380
I20260926 13:40:12.539731 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:40:12.540127 1220 external_mini_cluster.cc:1428] Setting key ae5fee17851898f5f73a7d0c25e52455
Sep 26 13:40:12 dist-test-slave-1wns krb5kdc[5274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430012, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:12.546936 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:12.531447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:42811 (local address 127.1.49.62:43391)
0926 13:40:12.531611 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:12.531620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:12.532632 (+ 1012us) server_negotiation.cc:407] Connection header received
0926 13:40:12.534238 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:12.534242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:12.534307 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:12.534415 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:12.535713 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:12.536372 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:12.537620 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:12.537867 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:12.537979 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:12.541493 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:12.541498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:12.541500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:12.541527 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:12.543670 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:12.544641 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:12.544653 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:12.544655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:12.544783 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:12.545310 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:12.545316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:12.545317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:12.545601 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:12.545722 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:12.546375 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 13:40:12.546639 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 13:40:12.547948 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:12.531916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:43391 (local address 127.1.49.1:42811)
0926 13:40:12.532443 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:12.532493 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:12.533989 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:12.534607 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:12.534614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:12.535010 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:12.535542 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:12.535556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:12.536526 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:12.536529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:12.537465 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:12.537475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:12.538047 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:12.538051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:12.538249 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:12.538938 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:12.538956 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:12.541150 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:12.543874 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:12.543887 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:12.543914 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:12.544440 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:12.544955 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:12.544958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:12.544959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:12.545180 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:12.545745 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:12.545752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:12.546081 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 13:40:12.546265 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 13:40:12.546604 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":116,"threads_started":1}
I20260926 13:40:12.549686 5508 heartbeater.cc:347] Connected to a master server at 127.1.49.62:43391
I20260926 13:40:12.550021 5508 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:12.550689 5508 heartbeater.cc:511] Master 127.1.49.62:43391 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
I20260926 13:40:12.552728 5318 ts_manager.cc:194] Registered new tserver with Master: 3861dfe019934bee8b2121a538493020 (127.1.49.1:36553)
I20260926 13:40:12.554075 5318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:42811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:40:12.576555 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:43391
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:12.698386 5513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:12.698779 5513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:12.698894 5513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:12.703922 5513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:12.704046 5513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:12.704151 5513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:40:12.710049 5513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:43391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:12.711274 5513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:12.712477 5513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:12.723135 5521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:12.723323 5519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:12.723135 5518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:12.723191 5513 server_base.cc:1034] running on GCE node
I20260926 13:40:12.724184 5513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:12.725015 5513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:12.726228 5513 hybrid_clock.cc:648] HybridClock initialized: now 1790430012726211 us; error 38 us; skew 500 ppm
Sep 26 13:40:12 dist-test-slave-1wns krb5kdc[5274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430012, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:12.729592 5513 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:40:12.730470 5513 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:12.730578 5513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:12.730834 5513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:12.733265 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "49fe41cdcc304594a211e9f3bc5f296e"
format_stamp: "Formatted at 2026-09-26 13:40:12 on dist-test-slave-1wns"
server_key: "276b0fb532f99faecd2f6333bc36cc5e"
server_key_iv: "be002e4c61f87186d2e2972f04cbf3f2"
server_key_version: "encryptionkey@0"
I20260926 13:40:12.733916 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "49fe41cdcc304594a211e9f3bc5f296e"
format_stamp: "Formatted at 2026-09-26 13:40:12 on dist-test-slave-1wns"
server_key: "276b0fb532f99faecd2f6333bc36cc5e"
server_key_iv: "be002e4c61f87186d2e2972f04cbf3f2"
server_key_version: "encryptionkey@0"
I20260926 13:40:12.738404 5513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:40:12.741741 5527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:12.743333 5513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:40:12.743578 5513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "49fe41cdcc304594a211e9f3bc5f296e"
format_stamp: "Formatted at 2026-09-26 13:40:12 on dist-test-slave-1wns"
server_key: "276b0fb532f99faecd2f6333bc36cc5e"
server_key_iv: "be002e4c61f87186d2e2972f04cbf3f2"
server_key_version: "encryptionkey@0"
I20260926 13:40:12.743774 5513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:12.764997 5513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:12.768869 5513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:12.769130 5513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:12.769807 5513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:12.770780 5513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:12.770860 5513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:12.770954 5513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:12.771013 5513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:12.781417 5513 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:38617
I20260926 13:40:12.781455 5640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:38617 every 8 connection(s)
I20260926 13:40:12.782112 5513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:40:12.784653 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 5513
I20260926 13:40:12.784777 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:40:12.785179 1220 external_mini_cluster.cc:1428] Setting key 0d41259f18d3b584e7054919961ce674
Sep 26 13:40:12 dist-test-slave-1wns krb5kdc[5274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430012, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
I20260926 13:40:12.798431 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:12.784233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:59645 (local address 127.1.49.62:43391)
0926 13:40:12.784373 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:12.784377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:12.785753 (+ 1376us) server_negotiation.cc:407] Connection header received
0926 13:40:12.787483 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:12.787487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:12.787560 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:12.787670 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:12.789036 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:12.789601 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:12.790866 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:12.791112 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:12.791236 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:12.794105 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:12.794108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:12.794111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:12.794139 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:12.795977 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:12.796577 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:12.796581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:12.796583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:12.796670 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:12.797059 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:12.797062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:12.797064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:12.797270 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:12.797511 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:12.798095 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 13:40:12.798244 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
I20260926 13:40:12.799479 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:12.784877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:43391 (local address 127.1.49.2:59645)
0926 13:40:12.785526 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:12.785605 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:12.787242 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:12.787884 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:12.787892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:12.788293 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:12.788854 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:12.788869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:12.789746 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:12.789750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:12.790723 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:12.790734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:12.791289 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:12.791294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:12.791484 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:12.792238 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:12.792256 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:12.793832 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:12.796125 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:12.796132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:12.796154 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:12.796460 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:12.796775 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:12.796778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:12.796781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:12.796948 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:12.797511 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:12.797521 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:12.797829 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 13:40:12.798050 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 13:40:12.798437 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":177,"threads_started":1}
I20260926 13:40:12.801092 5641 heartbeater.cc:347] Connected to a master server at 127.1.49.62:43391
I20260926 13:40:12.801396 5641 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:12.802006 5641 heartbeater.cc:511] Master 127.1.49.62:43391 requested a full tablet report, sending...
I20260926 13:40:12.803256 5318 ts_manager.cc:194] Registered new tserver with Master: 49fe41cdcc304594a211e9f3bc5f296e (127.1.49.2:38617)
I20260926 13:40:12.804018 5318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:59645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:40:12.821278 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:43391
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:12.943001 5646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:12.943356 5646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:12.943437 5646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:12.948135 5646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:12.948398 5646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:12.948668 5646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:40:12.954736 5646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:43391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:12.956058 5646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:12.957234 5646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:12.968232 5652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:12.968242 5651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:12.968252 5654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:12.968469 5646 server_base.cc:1034] running on GCE node
I20260926 13:40:12.969131 5646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:12.969991 5646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:12.971218 5646 hybrid_clock.cc:648] HybridClock initialized: now 1790430012971164 us; error 70 us; skew 500 ppm
Sep 26 13:40:12 dist-test-slave-1wns krb5kdc[5274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430012, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:12.974684 5646 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:40:12.975553 5646 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:12.975653 5646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:12.975970 5646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:12.978351 5646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "0342528111ce445ab85a4b426700a28c"
format_stamp: "Formatted at 2026-09-26 13:40:12 on dist-test-slave-1wns"
server_key: "3c45003edf84df2b72d09d3fe06184b5"
server_key_iv: "619c89a2531a002d02474793e785e00b"
server_key_version: "encryptionkey@0"
I20260926 13:40:12.979000 5646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "0342528111ce445ab85a4b426700a28c"
format_stamp: "Formatted at 2026-09-26 13:40:12 on dist-test-slave-1wns"
server_key: "3c45003edf84df2b72d09d3fe06184b5"
server_key_iv: "619c89a2531a002d02474793e785e00b"
server_key_version: "encryptionkey@0"
I20260926 13:40:12.983297 5646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 13:40:12.986302 5660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:12.987977 5646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 13:40:12.988147 5646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "0342528111ce445ab85a4b426700a28c"
format_stamp: "Formatted at 2026-09-26 13:40:12 on dist-test-slave-1wns"
server_key: "3c45003edf84df2b72d09d3fe06184b5"
server_key_iv: "619c89a2531a002d02474793e785e00b"
server_key_version: "encryptionkey@0"
I20260926 13:40:12.988353 5646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:13.002651 5646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:13.006671 5646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:13.006908 5646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:13.007606 5646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:13.008620 5646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:13.008689 5646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:13.008736 5646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:13.008752 5646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:13.018348 5646 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:40433
I20260926 13:40:13.018368 5773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:40433 every 8 connection(s)
I20260926 13:40:13.019109 5646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:40:13.029181 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 5646
I20260926 13:40:13.029366 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:40:13.029839 1220 external_mini_cluster.cc:1428] Setting key 166f2a14f5aef50158fab715ca4bae9f
Sep 26 13:40:13 dist-test-slave-1wns krb5kdc[5274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430012, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:13.036330 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.021026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:53251 (local address 127.1.49.62:43391)
0926 13:40:13.021191 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:13.021196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:13.022188 (+ 992us) server_negotiation.cc:407] Connection header received
0926 13:40:13.023673 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:13.023676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:13.023730 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:13.023896 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:13.025556 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.026178 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.027636 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.028030 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.028131 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:13.031971 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:13.031975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:13.031978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:13.032011 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:13.033886 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:13.034536 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:13.034539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:13.034541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:13.034625 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:13.035001 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:13.035004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:13.035005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:13.035206 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:13.035297 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:13.036074 (+ 777us) server_negotiation.cc:299] Negotiation successful
0926 13:40:13.036201 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 13:40:13.037513 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.021516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:43391 (local address 127.1.49.3:53251)
0926 13:40:13.022020 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:13.022067 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:13.023414 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:13.024123 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:13.024133 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:13.024703 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:13.025279 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:13.025296 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.026345 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.026348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:13.027458 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:13.027470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.028230 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.028234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:13.028510 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:13.029202 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:13.029232 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:13.031704 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:13.034032 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:13.034039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:13.034055 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:13.034365 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:13.034731 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:13.034734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:13.034735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:13.034902 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:13.035318 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:13.035324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:13.035706 (+ 382us) client_negotiation.cc:769] Sending connection context
0926 13:40:13.035994 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 13:40:13.036324 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
I20260926 13:40:13.039331 5774 heartbeater.cc:347] Connected to a master server at 127.1.49.62:43391
I20260926 13:40:13.039655 5774 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:13.040266 5774 heartbeater.cc:511] Master 127.1.49.62:43391 requested a full tablet report, sending...
I20260926 13:40:13.041667 5318 ts_manager.cc:194] Registered new tserver with Master: 0342528111ce445ab85a4b426700a28c (127.1.49.3:40433)
I20260926 13:40:13.042343 5318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:53251
I20260926 13:40:13.045341 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:40:13.056985 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.047990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59084 (local address 127.1.49.62:43391)
0926 13:40:13.048173 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:13.048190 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:13.048323 (+ 133us) server_negotiation.cc:407] Connection header received
0926 13:40:13.048530 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:13.048534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:13.048594 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:13.048697 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:13.049597 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.050134 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.051055 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.051282 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.051414 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:13.053134 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:13.053138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:13.053141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:13.053179 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:13.055054 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:13.055588 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:13.055591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:13.055593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:13.055689 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:13.056022 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:13.056025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:13.056027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:13.056283 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:13.056435 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:13.056690 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 13:40:13.056814 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 13:40:13.060923 5318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:40:13.064708 5318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:40:13.081959 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.075203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.1.49.3:40433)
0926 13:40:13.076585 (+ 1382us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:13.076599 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:13.076621 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:40:13.076700 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:13.076705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:13.076929 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:13.077093 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:13.077423 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.078644 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.080304 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.081153 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.081266 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:13.081426 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:13.081669 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 13:40:13.081784 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1278,"thread_start_us":191,"threads_started":1}
I20260926 13:40:13.082166 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.075404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:36553 (local address 127.0.0.1:49180)
0926 13:40:13.076343 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:13.076358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:13.076439 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:13.077272 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:13.077274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:13.077289 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:13.077394 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:13.077402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.078787 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.078795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:13.080620 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:13.080634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.081742 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.081746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:13.081843 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:13.081870 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 13:40:13.081918 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 13:40:13.082001 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":156,"thread_start_us":80,"threads_started":1}
I20260926 13:40:13.081981 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.075017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:40433 (local address 127.0.0.1:52998)
0926 13:40:13.076098 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:13.076158 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:13.076301 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:13.077108 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:13.077114 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:13.077137 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:13.077296 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:13.077303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.078806 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.078809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:13.080172 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:13.080182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.081292 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.081296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:13.081431 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:13.081480 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 13:40:13.081616 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 13:40:13.081779 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":308,"thread_start_us":98,"threads_started":1}
I20260926 13:40:13.082500 5788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.075200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:38617 (local address 127.0.0.1:45798)
0926 13:40:13.076247 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:13.076258 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:13.076325 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:13.077421 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:13.077423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:13.077437 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:13.077548 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:13.077554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.078890 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.078893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:13.080757 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:13.080767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.081991 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.081998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:13.082124 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:13.082154 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 13:40:13.082237 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 13:40:13.082333 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":149,"thread_start_us":106,"threads_started":1}
I20260926 13:40:13.082940 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.075970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45798 (local address 127.1.49.2:38617)
0926 13:40:13.076697 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:13.076706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:13.076724 (+ 18us) server_negotiation.cc:407] Connection header received
0926 13:40:13.076785 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:13.076789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:13.076940 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:13.077107 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:13.077673 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.078772 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.080891 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.081680 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.081784 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:13.081925 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:13.082724 (+ 799us) server_negotiation.cc:299] Negotiation successful
0926 13:40:13.082818 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":97,"threads_started":1}
I20260926 13:40:13.082939 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.075878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.1.49.1:36553)
0926 13:40:13.076585 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:13.076595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:13.076621 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:40:13.076700 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:13.076705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:13.076929 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:13.077093 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:13.077529 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.078624 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.080771 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.081578 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.082502 (+ 924us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:13.082655 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:13.082712 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 13:40:13.082809 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":87,"threads_started":1}
I20260926 13:40:13.085824 5708 tablet_service.cc:1467] Processing CreateTablet for tablet 62e251c23a65408696c404bc3e32992a (DEFAULT_TABLE table=test-table [id=34fa4794682d4b888540830e516b5c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:13.085829 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 62e251c23a65408696c404bc3e32992a (DEFAULT_TABLE table=test-table [id=34fa4794682d4b888540830e516b5c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:13.085827 5575 tablet_service.cc:1467] Processing CreateTablet for tablet 62e251c23a65408696c404bc3e32992a (DEFAULT_TABLE table=test-table [id=34fa4794682d4b888540830e516b5c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:13.086951 5708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e251c23a65408696c404bc3e32992a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:13.086946 5575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e251c23a65408696c404bc3e32992a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:13.086946 5442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e251c23a65408696c404bc3e32992a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:13.094660 5793 tablet_bootstrap.cc:492] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c: Bootstrap starting.
I20260926 13:40:13.096076 5791 tablet_bootstrap.cc:492] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020: Bootstrap starting.
I20260926 13:40:13.097285 5793 tablet_bootstrap.cc:654] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:13.098114 5791 tablet_bootstrap.cc:654] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:13.098567 5793 log.cc:824] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:13.098912 5791 log.cc:824] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:13.099498 5792 tablet_bootstrap.cc:492] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e: Bootstrap starting.
I20260926 13:40:13.101668 5793 tablet_bootstrap.cc:492] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c: No bootstrap required, opened a new log
I20260926 13:40:13.101929 5793 ts_tablet_manager.cc:1397] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.007s
I20260926 13:40:13.102406 5792 tablet_bootstrap.cc:654] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:13.103330 5791 tablet_bootstrap.cc:492] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020: No bootstrap required, opened a new log
I20260926 13:40:13.103574 5791 ts_tablet_manager.cc:1397] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260926 13:40:13.103902 5792 log.cc:824] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:13.106447 5792 tablet_bootstrap.cc:492] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e: No bootstrap required, opened a new log
I20260926 13:40:13.106688 5792 ts_tablet_manager.cc:1397] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20260926 13:40:13.106503 5793 raft_consensus.cc:348] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0342528111ce445ab85a4b426700a28c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40433 } } peers { permanent_uuid: "49fe41cdcc304594a211e9f3bc5f296e" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 38617 } } peers { permanent_uuid: "3861dfe019934bee8b2121a538493020" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 36553 } }
I20260926 13:40:13.106849 5793 raft_consensus.cc:374] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:13.106918 5793 raft_consensus.cc:729] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0342528111ce445ab85a4b426700a28c, State: Initialized, Role: FOLLOWER
I20260926 13:40:13.107503 5793 consensus_queue.cc:260] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0342528111ce445ab85a4b426700a28c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40433 } } peers { permanent_uuid: "49fe41cdcc304594a211e9f3bc5f296e" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 38617 } } peers { permanent_uuid: "3861dfe019934bee8b2121a538493020" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 36553 } }
I20260926 13:40:13.108560 5791 raft_consensus.cc:348] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0342528111ce445ab85a4b426700a28c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40433 } } peers { permanent_uuid: "49fe41cdcc304594a211e9f3bc5f296e" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 38617 } } peers { permanent_uuid: "3861dfe019934bee8b2121a538493020" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 36553 } }
I20260926 13:40:13.108799 5793 ts_tablet_manager.cc:1428] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260926 13:40:13.108861 5791 raft_consensus.cc:374] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:13.108935 5791 raft_consensus.cc:729] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3861dfe019934bee8b2121a538493020, State: Initialized, Role: FOLLOWER
I20260926 13:40:13.109274 5774 heartbeater.cc:503] Master 127.1.49.62:43391 was elected leader, sending a full tablet report...
I20260926 13:40:13.109473 5791 consensus_queue.cc:260] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0342528111ce445ab85a4b426700a28c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40433 } } peers { permanent_uuid: "49fe41cdcc304594a211e9f3bc5f296e" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 38617 } } peers { permanent_uuid: "3861dfe019934bee8b2121a538493020" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 36553 } }
I20260926 13:40:13.110461 5508 heartbeater.cc:503] Master 127.1.49.62:43391 was elected leader, sending a full tablet report...
I20260926 13:40:13.110632 5791 ts_tablet_manager.cc:1428] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20260926 13:40:13.111161 5792 raft_consensus.cc:348] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0342528111ce445ab85a4b426700a28c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40433 } } peers { permanent_uuid: "49fe41cdcc304594a211e9f3bc5f296e" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 38617 } } peers { permanent_uuid: "3861dfe019934bee8b2121a538493020" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 36553 } }
I20260926 13:40:13.111454 5792 raft_consensus.cc:374] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:13.111532 5792 raft_consensus.cc:729] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fe41cdcc304594a211e9f3bc5f296e, State: Initialized, Role: FOLLOWER
I20260926 13:40:13.112251 5792 consensus_queue.cc:260] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0342528111ce445ab85a4b426700a28c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40433 } } peers { permanent_uuid: "49fe41cdcc304594a211e9f3bc5f296e" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 38617 } } peers { permanent_uuid: "3861dfe019934bee8b2121a538493020" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 36553 } }
I20260926 13:40:13.113482 5641 heartbeater.cc:503] Master 127.1.49.62:43391 was elected leader, sending a full tablet report...
I20260926 13:40:13.113847 5792 ts_tablet_manager.cc:1428] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260926 13:40:13.125911 5798 raft_consensus.cc:482] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:40:13.126194 5798 raft_consensus.cc:504] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0342528111ce445ab85a4b426700a28c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40433 } } peers { permanent_uuid: "49fe41cdcc304594a211e9f3bc5f296e" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 38617 } } peers { permanent_uuid: "3861dfe019934bee8b2121a538493020" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 36553 } }
I20260926 13:40:13.127859 5798 leader_election.cc:288] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0342528111ce445ab85a4b426700a28c (127.1.49.3:40433), 49fe41cdcc304594a211e9f3bc5f296e (127.1.49.2:38617)
I20260926 13:40:13.132750 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.128249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:58035 (local address 127.1.49.2:38617)
0926 13:40:13.128449 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:13.128454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:13.128641 (+ 187us) server_negotiation.cc:407] Connection header received
0926 13:40:13.128821 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:13.128824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:13.128890 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:13.128988 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:13.129392 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.130060 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.131529 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.132021 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.132108 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:13.132169 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:13.132487 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 13:40:13.132577 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 13:40:13.132920 5800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.128121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:40433 (local address 127.1.49.1:51307)
0926 13:40:13.128662 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:13.128678 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:13.128826 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:13.129275 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:13.129277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:13.129294 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:13.129528 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:13.129536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.130528 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.130530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:13.131426 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:13.131465 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.132475 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.132479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:13.132584 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:13.132607 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:40:13.132691 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 13:40:13.132756 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":1,"thread_start_us":136,"threads_started":1}
I20260926 13:40:13.132964 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.128543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:51307 (local address 127.1.49.3:40433)
0926 13:40:13.128696 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:13.128700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:13.128797 (+ 97us) server_negotiation.cc:407] Connection header received
0926 13:40:13.129057 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:13.129060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:13.129113 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:13.129208 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:13.129693 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.130393 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.131626 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.132254 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.132382 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:13.132473 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:13.132731 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 13:40:13.132821 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 13:40:13.132817 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.128106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:38617 (local address 127.1.49.1:58035)
0926 13:40:13.128518 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:13.128531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:13.128684 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:13.129057 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:13.129060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:13.129080 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:13.129246 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:13.129254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.130196 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.130200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:13.131372 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:13.131382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.132171 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.132183 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:13.132315 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:13.132361 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 13:40:13.132443 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 13:40:13.132610 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":26,"thread_start_us":131,"threads_started":1}
I20260926 13:40:13.133801 5728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e251c23a65408696c404bc3e32992a" candidate_uuid: "3861dfe019934bee8b2121a538493020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0342528111ce445ab85a4b426700a28c" is_pre_election: true
I20260926 13:40:13.134228 5728 raft_consensus.cc:2454] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3861dfe019934bee8b2121a538493020 in term 0.
I20260926 13:40:13.134212 5595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e251c23a65408696c404bc3e32992a" candidate_uuid: "3861dfe019934bee8b2121a538493020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fe41cdcc304594a211e9f3bc5f296e" is_pre_election: true
I20260926 13:40:13.134583 5595 raft_consensus.cc:2454] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3861dfe019934bee8b2121a538493020 in term 0.
I20260926 13:40:13.134859 5398 leader_election.cc:302] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0342528111ce445ab85a4b426700a28c, 3861dfe019934bee8b2121a538493020; no voters:
I20260926 13:40:13.135219 5798 raft_consensus.cc:2781] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:40:13.135306 5798 raft_consensus.cc:482] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:40:13.135362 5798 raft_consensus.cc:3037] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:13.136612 5798 raft_consensus.cc:504] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0342528111ce445ab85a4b426700a28c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40433 } } peers { permanent_uuid: "49fe41cdcc304594a211e9f3bc5f296e" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 38617 } } peers { permanent_uuid: "3861dfe019934bee8b2121a538493020" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 36553 } }
I20260926 13:40:13.137288 5798 leader_election.cc:288] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [CANDIDATE]: Term 1 election: Requested vote from peers 0342528111ce445ab85a4b426700a28c (127.1.49.3:40433), 49fe41cdcc304594a211e9f3bc5f296e (127.1.49.2:38617)
I20260926 13:40:13.137895 5728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e251c23a65408696c404bc3e32992a" candidate_uuid: "3861dfe019934bee8b2121a538493020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0342528111ce445ab85a4b426700a28c"
I20260926 13:40:13.137873 5595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e251c23a65408696c404bc3e32992a" candidate_uuid: "3861dfe019934bee8b2121a538493020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49fe41cdcc304594a211e9f3bc5f296e"
I20260926 13:40:13.138041 5728 raft_consensus.cc:3037] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:13.138042 5595 raft_consensus.cc:3037] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:13.139624 5728 raft_consensus.cc:2454] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3861dfe019934bee8b2121a538493020 in term 1.
I20260926 13:40:13.139626 5595 raft_consensus.cc:2454] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3861dfe019934bee8b2121a538493020 in term 1.
I20260926 13:40:13.140136 5398 leader_election.cc:302] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0342528111ce445ab85a4b426700a28c, 3861dfe019934bee8b2121a538493020; no voters:
I20260926 13:40:13.140427 5798 raft_consensus.cc:2781] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:13.140832 5798 raft_consensus.cc:686] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [term 1 LEADER]: Becoming Leader. State: Replica: 3861dfe019934bee8b2121a538493020, State: Running, Role: LEADER
I20260926 13:40:13.141275 5798 consensus_queue.cc:237] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0342528111ce445ab85a4b426700a28c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40433 } } peers { permanent_uuid: "49fe41cdcc304594a211e9f3bc5f296e" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 38617 } } peers { permanent_uuid: "3861dfe019934bee8b2121a538493020" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 36553 } }
I20260926 13:40:13.145092 5316 catalog_manager.cc:5495] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3861dfe019934bee8b2121a538493020 (127.1.49.1). New cstate: current_term: 1 leader_uuid: "3861dfe019934bee8b2121a538493020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0342528111ce445ab85a4b426700a28c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49fe41cdcc304594a211e9f3bc5f296e" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 38617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3861dfe019934bee8b2121a538493020" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 36553 } health_report { overall_health: HEALTHY } } }
I20260926 13:40:13.167155 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.162883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.1.49.1:36553)
0926 13:40:13.163053 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:13.163057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:13.163071 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:40:13.163149 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:13.163151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:13.163209 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:13.163314 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:40:13.163703 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.164444 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.165466 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:13.165683 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:13.165771 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:13.166140 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:40:13.166676 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:40:13.166793 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:13.166918 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 13:40:13.167015 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 13:40:13.174265 5728 raft_consensus.cc:1261] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c [term 1 FOLLOWER]: Refusing update from remote peer 3861dfe019934bee8b2121a538493020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:13.174265 5595 raft_consensus.cc:1261] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e [term 1 FOLLOWER]: Refusing update from remote peer 3861dfe019934bee8b2121a538493020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:13.175164 5798 consensus_queue.cc:1035] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49fe41cdcc304594a211e9f3bc5f296e" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 38617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:13.175395 5802 consensus_queue.cc:1035] T 62e251c23a65408696c404bc3e32992a P 3861dfe019934bee8b2121a538493020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0342528111ce445ab85a4b426700a28c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:13.186033 5806 mvcc.cc:205] Tried to move back new op lower bound from 7333601333952815104 to 7333601333830189056. Current Snapshot: MvccSnapshot[applied={T|T < 7333601333952815104 or (T in {7333601333952815104})}]
I20260926 13:40:13.186260 5807 mvcc.cc:205] Tried to move back new op lower bound from 7333601333952815104 to 7333601333830189056. Current Snapshot: MvccSnapshot[applied={T|T < 7333601333952815104}]
I20260926 13:40:13.194526 5318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:13.194722 5318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:13.197606 5318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec0d73a629b340dfa1ecf87c673f5fb1: Sending DeleteTablet for 3 replicas of tablet 62e251c23a65408696c404bc3e32992a
I20260926 13:40:13.198661 5575 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e251c23a65408696c404bc3e32992a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:13 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:45798
I20260926 13:40:13.198766 5708 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e251c23a65408696c404bc3e32992a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:13 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:52998
I20260926 13:40:13.198757 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e251c23a65408696c404bc3e32992a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:13 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:49180
I20260926 13:40:13.199294 5816 tablet_replica.cc:330] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c: stopping tablet replica
I20260926 13:40:13.199638 5815 tablet_replica.cc:330] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e: stopping tablet replica
I20260926 13:40:13.199762 5816 raft_consensus.cc:2229] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:13.200068 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 5380
I20260926 13:40:13.200215 5815 raft_consensus.cc:2229] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:13.200611 5815 raft_consensus.cc:2258] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:13.201205 5816 raft_consensus.cc:2258] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:13.203557 5816 ts_tablet_manager.cc:1905] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:13.203554 5815 ts_tablet_manager.cc:1905] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:13.207031 5816 ts_tablet_manager.cc:1918] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:13.207053 5815 ts_tablet_manager.cc:1918] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:13.207190 5815 log.cc:1196] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/62e251c23a65408696c404bc3e32992a
I20260926 13:40:13.207190 5816 log.cc:1196] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/62e251c23a65408696c404bc3e32992a
I20260926 13:40:13.207551 5816 ts_tablet_manager.cc:1939] T 62e251c23a65408696c404bc3e32992a P 0342528111ce445ab85a4b426700a28c: Deleting consensus metadata
I20260926 13:40:13.207551 5815 ts_tablet_manager.cc:1939] T 62e251c23a65408696c404bc3e32992a P 49fe41cdcc304594a211e9f3bc5f296e: Deleting consensus metadata
W20260926 13:40:13.207762 5304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:36553 (error 108)
I20260926 13:40:13.208034 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 5513
I20260926 13:40:13.209585 5305 catalog_manager.cc:4841] TS 0342528111ce445ab85a4b426700a28c (127.1.49.3:40433): tablet 62e251c23a65408696c404bc3e32992a (table test-table [id=34fa4794682d4b888540830e516b5c68]) successfully deleted
I20260926 13:40:13.211102 5788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.210713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:36553 (local address 127.0.0.1:47752)
0926 13:40:13.210900 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:13.210992 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 13:40:13.211392 5304 catalog_manager.cc:4584] TS 3861dfe019934bee8b2121a538493020 (127.1.49.1:36553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62e251c23a65408696c404bc3e32992a: Network error: Client connection negotiation failed: client connection to 127.1.49.1:36553: connect: Connection refused (error 111)
I20260926 13:40:13.214715 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 5646
I20260926 13:40:13.216349 5788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.214879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:38617 (local address 127.0.0.1:43862)
0926 13:40:13.216181 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:13.216249 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.2:38617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1204}
I20260926 13:40:13.222474 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 5287
2026-09-26T13:40:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2489 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T13:40:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:13Z Disabled control of system clock
I20260926 13:40:13.255790 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:45389
--webserver_interface=127.1.49.62
--webserver_port=0
--builtin_ntp_servers=127.1.49.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:45389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 13:40:13.381323 5822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:13.381764 5822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:13.381852 5822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:13.386211 5822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:13.386296 5822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:13.386312 5822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:13.386329 5822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:13.386346 5822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:13.391633 5822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:45389
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:45389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:13.392936 5822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:13.394363 5822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:13.403868 5827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:13.404026 5828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:13.403868 5830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:13.404000 5822 server_base.cc:1034] running on GCE node
I20260926 13:40:13.404907 5822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:13.406157 5822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:13.407398 5822 hybrid_clock.cc:648] HybridClock initialized: now 1790430013407352 us; error 57 us; skew 500 ppm
I20260926 13:40:13.409828 5822 webserver.cc:460] Webserver started at http://127.1.49.62:33551/ using document root <none> and password file <none>
I20260926 13:40:13.410491 5822 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:13.410597 5822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:13.410873 5822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:13.413237 5822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "48b90f1e9b0b488ba3fb6e5e6e84a27f"
format_stamp: "Formatted at 2026-09-26 13:40:13 on dist-test-slave-1wns"
server_key: "ffdd0b239751b26428f17b3ce89b698c"
server_key_iv: "a35078491144c285648bba5d5a004c19"
server_key_version: "encryptionkey@0"
I20260926 13:40:13.413867 5822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "48b90f1e9b0b488ba3fb6e5e6e84a27f"
format_stamp: "Formatted at 2026-09-26 13:40:13 on dist-test-slave-1wns"
server_key: "ffdd0b239751b26428f17b3ce89b698c"
server_key_iv: "a35078491144c285648bba5d5a004c19"
server_key_version: "encryptionkey@0"
I20260926 13:40:13.418279 5822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:40:13.421554 5836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:13.423126 5822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:40:13.423313 5822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "48b90f1e9b0b488ba3fb6e5e6e84a27f"
format_stamp: "Formatted at 2026-09-26 13:40:13 on dist-test-slave-1wns"
server_key: "ffdd0b239751b26428f17b3ce89b698c"
server_key_iv: "a35078491144c285648bba5d5a004c19"
server_key_version: "encryptionkey@0"
I20260926 13:40:13.423470 5822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:13.429051 5822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:13.429814 5822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:13.430047 5822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:13.438156 5822 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:45389
I20260926 13:40:13.438171 5888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:45389 every 8 connection(s)
I20260926 13:40:13.439222 5822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:13.442816 5889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:13.443300 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 5822
I20260926 13:40:13.443424 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:13.443730 1220 external_mini_cluster.cc:1428] Setting key d5f72109bd7b984e02db5116c2b143a6
I20260926 13:40:13.449798 5889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f: Bootstrap starting.
I20260926 13:40:13.449862 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.445281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.1.49.62:45389)
0926 13:40:13.445941 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:13.445956 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:13.445993 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:40:13.446653 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:13.446669 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:13.447049 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:13.447366 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:13.448279 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:13.448284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:13.448296 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:13.448306 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:13.448364 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:13.448381 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:13.448500 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:13.448829 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 13:40:13.449113 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":138,"threads_started":1}
I20260926 13:40:13.452492 5889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:13.453275 5889 log.cc:824] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:13.455487 5889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f: No bootstrap required, opened a new log
I20260926 13:40:13.459414 5889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b90f1e9b0b488ba3fb6e5e6e84a27f" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 45389 } }
I20260926 13:40:13.459733 5889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:13.459836 5889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b90f1e9b0b488ba3fb6e5e6e84a27f, State: Initialized, Role: FOLLOWER
I20260926 13:40:13.460296 5889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b90f1e9b0b488ba3fb6e5e6e84a27f" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 45389 } }
I20260926 13:40:13.460449 5889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:13.460549 5889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:13.460680 5889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:13.461681 5889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b90f1e9b0b488ba3fb6e5e6e84a27f" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 45389 } }
I20260926 13:40:13.462059 5889 leader_election.cc:302] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48b90f1e9b0b488ba3fb6e5e6e84a27f; no voters:
I20260926 13:40:13.462419 5889 leader_election.cc:288] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:13.462610 5894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:13.462864 5894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [term 1 LEADER]: Becoming Leader. State: Replica: 48b90f1e9b0b488ba3fb6e5e6e84a27f, State: Running, Role: LEADER
I20260926 13:40:13.463203 5894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b90f1e9b0b488ba3fb6e5e6e84a27f" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 45389 } }
I20260926 13:40:13.463797 5889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:13.464890 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48b90f1e9b0b488ba3fb6e5e6e84a27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b90f1e9b0b488ba3fb6e5e6e84a27f" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 45389 } } }
I20260926 13:40:13.465046 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:13.465062 5896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48b90f1e9b0b488ba3fb6e5e6e84a27f. Latest consensus state: current_term: 1 leader_uuid: "48b90f1e9b0b488ba3fb6e5e6e84a27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b90f1e9b0b488ba3fb6e5e6e84a27f" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 45389 } } }
I20260926 13:40:13.465126 5896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:13.465407 5903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:13.468715 5903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:13.474658 5903 catalog_manager.cc:1347] Generated new cluster ID: 16f9b8e98f9a4c11a6c6b51c06647cf9
I20260926 13:40:13.474748 5903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:13.492267 5903 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:13.492671 5903 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:13.499389 5903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f: Generated new TSK 0
I20260926 13:40:13.500406 5903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:40:13.510519 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:45389
--builtin_ntp_servers=127.1.49.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 13:40:13.631531 5913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:13.631937 5913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:13.632026 5913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:13.636426 5913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:13.636549 5913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:13.636662 5913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:40:13.641893 5913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:45389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:13.643131 5913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:13.644769 5913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:13.655578 5918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:13.655679 5919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:13.655680 5921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:13.656049 5913 server_base.cc:1034] running on GCE node
I20260926 13:40:13.656648 5913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:13.657467 5913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:13.658680 5913 hybrid_clock.cc:648] HybridClock initialized: now 1790430013658641 us; error 58 us; skew 500 ppm
I20260926 13:40:13.660845 5913 webserver.cc:460] Webserver started at http://127.1.49.1:36463/ using document root <none> and password file <none>
I20260926 13:40:13.661481 5913 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:13.661592 5913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:13.661844 5913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:13.664224 5913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "06e012afdd5c40188a4a850baf97c5bb"
format_stamp: "Formatted at 2026-09-26 13:40:13 on dist-test-slave-1wns"
server_key: "a370eb712fc7cd85a7b7c39cc9e5c40c"
server_key_iv: "2980baf429ab2cfdbcac5a5e37ac1758"
server_key_version: "encryptionkey@0"
I20260926 13:40:13.664863 5913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "06e012afdd5c40188a4a850baf97c5bb"
format_stamp: "Formatted at 2026-09-26 13:40:13 on dist-test-slave-1wns"
server_key: "a370eb712fc7cd85a7b7c39cc9e5c40c"
server_key_iv: "2980baf429ab2cfdbcac5a5e37ac1758"
server_key_version: "encryptionkey@0"
I20260926 13:40:13.669162 5913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:40:13.672163 5927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:13.673666 5913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 13:40:13.673851 5913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "06e012afdd5c40188a4a850baf97c5bb"
format_stamp: "Formatted at 2026-09-26 13:40:13 on dist-test-slave-1wns"
server_key: "a370eb712fc7cd85a7b7c39cc9e5c40c"
server_key_iv: "2980baf429ab2cfdbcac5a5e37ac1758"
server_key_version: "encryptionkey@0"
I20260926 13:40:13.674003 5913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:13.679464 5913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:13.680294 5913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:13.680517 5913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:13.681167 5913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:13.682497 5913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:13.682580 5913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:13.682660 5913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:13.682709 5913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:13.692909 5913 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:46673
I20260926 13:40:13.692979 6040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:46673 every 8 connection(s)
I20260926 13:40:13.694060 5913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:40:13.697960 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 5913
I20260926 13:40:13.698132 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:40:13.698436 1220 external_mini_cluster.cc:1428] Setting key 895ac15b05ede7af8d9de9b6e3cfee26
I20260926 13:40:13.700744 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:45389
--builtin_ntp_servers=127.1.49.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 13:40:13.701926 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.696562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:39265 (local address 127.1.49.62:45389)
0926 13:40:13.696734 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:13.696738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:13.697940 (+ 1202us) server_negotiation.cc:407] Connection header received
0926 13:40:13.699687 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:13.699691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:13.699730 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:13.699892 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:13.701289 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:13.701296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:13.701302 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:13.701328 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:13.701357 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:13.701371 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:13.701485 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:13.701681 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 13:40:13.701738 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 13:40:13.703054 6043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.696907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:45389 (local address 127.1.49.1:39265)
0926 13:40:13.697717 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:13.697765 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:13.699424 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:13.700121 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:13.700133 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:13.700260 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:13.700950 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:40:13.700952 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:13.701022 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:40:13.701122 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:13.701506 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:13.701514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:13.701520 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:40:13.701703 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 13:40:13.701948 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":152,"threads_started":1}
I20260926 13:40:13.704699 6041 heartbeater.cc:347] Connected to a master server at 127.1.49.62:45389
I20260926 13:40:13.705020 6041 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:13.705525 6041 heartbeater.cc:511] Master 127.1.49.62:45389 requested a full tablet report, sending...
I20260926 13:40:13.707690 5853 ts_manager.cc:194] Registered new tserver with Master: 06e012afdd5c40188a4a850baf97c5bb (127.1.49.1:46673)
W20260926 13:40:13.828433 6044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:13.828822 6044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:13.828874 6044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:13.833776 6044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:13.833860 6044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:13.833930 6044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:40:13.839303 6044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:45389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:13.840467 6044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:13.841900 6044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:13.852677 6052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:13.852732 6049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:13.852712 6044 server_base.cc:1034] running on GCE node
W20260926 13:40:13.852833 6050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:13.853328 6044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:13.854205 6044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:13.855405 6044 hybrid_clock.cc:648] HybridClock initialized: now 1790430013855374 us; error 52 us; skew 500 ppm
I20260926 13:40:13.858049 6044 webserver.cc:460] Webserver started at http://127.1.49.2:36597/ using document root <none> and password file <none>
I20260926 13:40:13.858822 6044 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:13.858901 6044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:13.859172 6044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:13.861655 6044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "c00031f4b21e419887ca18bce579052a"
format_stamp: "Formatted at 2026-09-26 13:40:13 on dist-test-slave-1wns"
server_key: "5dc9b03b0be3cdcfdae805d591c7c4af"
server_key_iv: "5d3bd9141760e54006d0eb37a427d5a2"
server_key_version: "encryptionkey@0"
I20260926 13:40:13.862323 6044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "c00031f4b21e419887ca18bce579052a"
format_stamp: "Formatted at 2026-09-26 13:40:13 on dist-test-slave-1wns"
server_key: "5dc9b03b0be3cdcfdae805d591c7c4af"
server_key_iv: "5d3bd9141760e54006d0eb37a427d5a2"
server_key_version: "encryptionkey@0"
I20260926 13:40:13.867156 6044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 13:40:13.870186 6058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:13.871668 6044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:40:13.871873 6044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "c00031f4b21e419887ca18bce579052a"
format_stamp: "Formatted at 2026-09-26 13:40:13 on dist-test-slave-1wns"
server_key: "5dc9b03b0be3cdcfdae805d591c7c4af"
server_key_iv: "5d3bd9141760e54006d0eb37a427d5a2"
server_key_version: "encryptionkey@0"
I20260926 13:40:13.872062 6044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:13.877102 6044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:13.877863 6044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:13.878065 6044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:13.878739 6044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:13.879850 6044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:13.879912 6044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:13.879989 6044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:13.880023 6044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:13.889212 6044 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:35929
I20260926 13:40:13.889307 6171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:35929 every 8 connection(s)
I20260926 13:40:13.890252 6044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:40:13.897814 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.892526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:57449 (local address 127.1.49.62:45389)
0926 13:40:13.892713 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:13.892718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:13.894006 (+ 1288us) server_negotiation.cc:407] Connection header received
0926 13:40:13.895636 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:13.895640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:13.895696 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:13.895796 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:13.897239 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:13.897244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:13.897247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:13.897263 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:13.897294 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:13.897304 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:13.897423 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:13.897595 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 13:40:13.897653 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 13:40:13.898885 6174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:13.892946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:45389 (local address 127.1.49.2:57449)
0926 13:40:13.893813 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:13.893862 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:13.895349 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:13.896069 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:13.896078 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:13.896192 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:13.896913 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:40:13.896915 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:13.896963 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:40:13.897054 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:13.897399 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:13.897409 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:13.897415 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:40:13.897641 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 13:40:13.897941 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":130,"threads_started":1}
I20260926 13:40:13.899258 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 6044
I20260926 13:40:13.899396 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:40:13.899910 1220 external_mini_cluster.cc:1428] Setting key 77e39a1121c9e7e5f0c22fffbbedee85
I20260926 13:40:13.900583 6172 heartbeater.cc:347] Connected to a master server at 127.1.49.62:45389
I20260926 13:40:13.900875 6172 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:13.901400 6172 heartbeater.cc:511] Master 127.1.49.62:45389 requested a full tablet report, sending...
I20260926 13:40:13.902325 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:45389
--builtin_ntp_servers=127.1.49.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 13:40:13.902760 5853 ts_manager.cc:194] Registered new tserver with Master: c00031f4b21e419887ca18bce579052a (127.1.49.2:35929)
W20260926 13:40:14.025940 6175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:14.026307 6175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:14.026388 6175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:14.031359 6175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:14.031478 6175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:14.031591 6175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:40:14.037088 6175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:45389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:14.038368 6175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:14.040102 6175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:14.051241 6180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:14.051223 6175 server_base.cc:1034] running on GCE node
W20260926 13:40:14.051203 6183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:14.051214 6181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:14.052076 6175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:14.052865 6175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:14.054085 6175 hybrid_clock.cc:648] HybridClock initialized: now 1790430014054056 us; error 36 us; skew 500 ppm
I20260926 13:40:14.056354 6175 webserver.cc:460] Webserver started at http://127.1.49.3:34503/ using document root <none> and password file <none>
I20260926 13:40:14.056996 6175 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:14.057096 6175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:14.057384 6175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:14.059636 6175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "97d79666974c4fb7a7e09627548f6a27"
format_stamp: "Formatted at 2026-09-26 13:40:14 on dist-test-slave-1wns"
server_key: "461bcf03101b3856155b5ebd45ff722e"
server_key_iv: "2810991df602ae99fdc7128f4c68792f"
server_key_version: "encryptionkey@0"
I20260926 13:40:14.060382 6175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "97d79666974c4fb7a7e09627548f6a27"
format_stamp: "Formatted at 2026-09-26 13:40:14 on dist-test-slave-1wns"
server_key: "461bcf03101b3856155b5ebd45ff722e"
server_key_iv: "2810991df602ae99fdc7128f4c68792f"
server_key_version: "encryptionkey@0"
I20260926 13:40:14.064845 6175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:40:14.067960 6189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:14.069756 6175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:14.069947 6175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "97d79666974c4fb7a7e09627548f6a27"
format_stamp: "Formatted at 2026-09-26 13:40:14 on dist-test-slave-1wns"
server_key: "461bcf03101b3856155b5ebd45ff722e"
server_key_iv: "2810991df602ae99fdc7128f4c68792f"
server_key_version: "encryptionkey@0"
I20260926 13:40:14.070094 6175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:14.075204 6175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:14.076017 6175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:14.076215 6175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:14.076881 6175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:14.077934 6175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:14.078008 6175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:14.078079 6175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:14.078130 6175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:14.087621 6175 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:34257
I20260926 13:40:14.087636 6302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:34257 every 8 connection(s)
I20260926 13:40:14.088687 6175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:40:14.090292 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 6175
I20260926 13:40:14.090462 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:40:14.090817 1220 external_mini_cluster.cc:1428] Setting key 6c31e5293a31127c3f7174976fd55804
I20260926 13:40:14.096376 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.090777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:35719 (local address 127.1.49.62:45389)
0926 13:40:14.090958 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:14.090963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:14.092428 (+ 1465us) server_negotiation.cc:407] Connection header received
0926 13:40:14.093902 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:14.093904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:14.093943 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:14.094023 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:14.095755 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:14.095759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:14.095761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:14.095775 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:14.095799 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:14.095865 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:14.095988 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:14.096208 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 13:40:14.096258 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 13:40:14.097517 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.091387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:45389 (local address 127.1.49.3:35719)
0926 13:40:14.092269 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:14.092325 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:14.093644 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:14.094298 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:14.094306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:14.094424 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:14.095278 (+ 854us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:40:14.095280 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:14.095329 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:40:14.095413 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:14.096019 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:14.096027 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:14.096033 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:40:14.096214 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 13:40:14.096455 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":159,"threads_started":1}
I20260926 13:40:14.099009 6303 heartbeater.cc:347] Connected to a master server at 127.1.49.62:45389
I20260926 13:40:14.099283 6303 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:14.099738 6303 heartbeater.cc:511] Master 127.1.49.62:45389 requested a full tablet report, sending...
I20260926 13:40:14.100977 5853 ts_manager.cc:194] Registered new tserver with Master: 97d79666974c4fb7a7e09627548f6a27 (127.1.49.3:34257)
I20260926 13:40:14.105618 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:40:14.109731 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.107892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.1.49.62:45389)
0926 13:40:14.108086 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:14.108090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:14.108227 (+ 137us) server_negotiation.cc:407] Connection header received
0926 13:40:14.108400 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:14.108403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:14.108441 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:14.108539 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:14.109298 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:14.109301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:14.109304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:14.109317 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:14.109340 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:14.109355 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:14.109452 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:14.109531 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 13:40:14.109577 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 13:40:14.113351 5853 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:40:14.116752 5853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:40:14.130988 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.127466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:35929 (local address 127.0.0.1:37502)
0926 13:40:14.128406 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:14.128421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:14.128500 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:14.129238 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:14.129245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:14.129377 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:14.130059 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:40:14.130061 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:14.130079 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:40:14.130142 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:14.130559 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:14.130568 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:14.130574 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:40:14.130723 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 13:40:14.130810 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"thread_start_us":140,"threads_started":1}
I20260926 13:40:14.130997 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.127592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.1.49.2:35929)
0926 13:40:14.128435 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:14.128457 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:14.128491 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:40:14.128671 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:14.128682 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:14.128951 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:14.129122 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:14.130290 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:14.130300 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:14.130313 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:14.130327 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:14.130396 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:14.130454 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:14.130576 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:14.130754 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 13:40:14.130831 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":148,"threads_started":1}
I20260926 13:40:14.131882 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.127346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:46673 (local address 127.0.0.1:34112)
0926 13:40:14.128119 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:14.128180 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:14.128302 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:14.130321 (+ 2019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:14.130325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:14.130341 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:14.131106 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:40:14.131108 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:14.131121 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:40:14.131150 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:14.131560 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:14.131564 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:14.131568 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:40:14.131662 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 13:40:14.131704 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":218,"threads_started":1}
I20260926 13:40:14.131992 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.128781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.1.49.1:46673)
0926 13:40:14.129244 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:14.129274 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:14.129300 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:40:14.129360 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:14.129365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:14.129595 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:14.129755 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:14.131305 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:14.131312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:14.131326 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:14.131340 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:14.131390 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:14.131429 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:14.131565 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:14.131678 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 13:40:14.131764 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":171,"threads_started":1}
I20260926 13:40:14.132953 6317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.128461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:34257 (local address 127.0.0.1:58158)
0926 13:40:14.129739 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:14.129755 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:14.129848 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:14.130891 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:14.130895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:14.130912 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:14.132103 (+ 1191us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:40:14.132105 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:14.132116 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:40:14.132141 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:14.132662 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:14.132666 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:14.132669 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:40:14.132750 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 13:40:14.132797 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"thread_start_us":108,"threads_started":1}
I20260926 13:40:14.133411 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.128785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58158 (local address 127.1.49.3:34257)
0926 13:40:14.129581 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:14.129593 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:14.129957 (+ 364us) server_negotiation.cc:407] Connection header received
0926 13:40:14.130017 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:14.130022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:14.130211 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:14.130342 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:14.132290 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:14.132297 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:14.132311 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:14.132327 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:14.132376 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:14.132437 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:14.132552 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:14.133179 (+ 627us) server_negotiation.cc:299] Negotiation successful
0926 13:40:14.133258 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":95,"threads_started":1}
I20260926 13:40:14.135098 6106 tablet_service.cc:1467] Processing CreateTablet for tablet 323d1ddcb52a4af8bb2b08a46f040ef6 (DEFAULT_TABLE table=test-table [id=4bb65827edaf44c9816be3aba0f60d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:14.135098 5975 tablet_service.cc:1467] Processing CreateTablet for tablet 323d1ddcb52a4af8bb2b08a46f040ef6 (DEFAULT_TABLE table=test-table [id=4bb65827edaf44c9816be3aba0f60d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:14.136439 6106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323d1ddcb52a4af8bb2b08a46f040ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:14.136451 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323d1ddcb52a4af8bb2b08a46f040ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:14.136523 6237 tablet_service.cc:1467] Processing CreateTablet for tablet 323d1ddcb52a4af8bb2b08a46f040ef6 (DEFAULT_TABLE table=test-table [id=4bb65827edaf44c9816be3aba0f60d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:14.137665 6237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323d1ddcb52a4af8bb2b08a46f040ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:14.146142 6322 tablet_bootstrap.cc:492] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb: Bootstrap starting.
I20260926 13:40:14.146723 6320 tablet_bootstrap.cc:492] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27: Bootstrap starting.
I20260926 13:40:14.148110 6321 tablet_bootstrap.cc:492] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a: Bootstrap starting.
I20260926 13:40:14.148648 6322 tablet_bootstrap.cc:654] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:14.149160 6320 tablet_bootstrap.cc:654] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:14.149495 6322 log.cc:824] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:14.150053 6320 log.cc:824] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:14.150605 6321 tablet_bootstrap.cc:654] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:14.151563 6321 log.cc:824] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:14.151772 6322 tablet_bootstrap.cc:492] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb: No bootstrap required, opened a new log
I20260926 13:40:14.152024 6322 ts_tablet_manager.cc:1397] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260926 13:40:14.152864 6320 tablet_bootstrap.cc:492] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27: No bootstrap required, opened a new log
I20260926 13:40:14.153070 6320 ts_tablet_manager.cc:1397] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:40:14.153770 6321 tablet_bootstrap.cc:492] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a: No bootstrap required, opened a new log
I20260926 13:40:14.153942 6321 ts_tablet_manager.cc:1397] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260926 13:40:14.157114 6320 raft_consensus.cc:348] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e012afdd5c40188a4a850baf97c5bb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 46673 } } peers { permanent_uuid: "c00031f4b21e419887ca18bce579052a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35929 } } peers { permanent_uuid: "97d79666974c4fb7a7e09627548f6a27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 34257 } }
I20260926 13:40:14.157176 6322 raft_consensus.cc:348] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e012afdd5c40188a4a850baf97c5bb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 46673 } } peers { permanent_uuid: "c00031f4b21e419887ca18bce579052a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35929 } } peers { permanent_uuid: "97d79666974c4fb7a7e09627548f6a27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 34257 } }
I20260926 13:40:14.157433 6320 raft_consensus.cc:374] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:14.157446 6322 raft_consensus.cc:374] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:14.157495 6322 raft_consensus.cc:729] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e012afdd5c40188a4a850baf97c5bb, State: Initialized, Role: FOLLOWER
I20260926 13:40:14.157495 6320 raft_consensus.cc:729] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d79666974c4fb7a7e09627548f6a27, State: Initialized, Role: FOLLOWER
I20260926 13:40:14.158001 6320 consensus_queue.cc:260] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e012afdd5c40188a4a850baf97c5bb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 46673 } } peers { permanent_uuid: "c00031f4b21e419887ca18bce579052a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35929 } } peers { permanent_uuid: "97d79666974c4fb7a7e09627548f6a27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 34257 } }
I20260926 13:40:14.158011 6322 consensus_queue.cc:260] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e012afdd5c40188a4a850baf97c5bb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 46673 } } peers { permanent_uuid: "c00031f4b21e419887ca18bce579052a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35929 } } peers { permanent_uuid: "97d79666974c4fb7a7e09627548f6a27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 34257 } }
I20260926 13:40:14.158569 6321 raft_consensus.cc:348] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e012afdd5c40188a4a850baf97c5bb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 46673 } } peers { permanent_uuid: "c00031f4b21e419887ca18bce579052a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35929 } } peers { permanent_uuid: "97d79666974c4fb7a7e09627548f6a27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 34257 } }
I20260926 13:40:14.159116 6321 raft_consensus.cc:374] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:14.159178 6321 raft_consensus.cc:729] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c00031f4b21e419887ca18bce579052a, State: Initialized, Role: FOLLOWER
I20260926 13:40:14.159201 6322 ts_tablet_manager.cc:1428] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260926 13:40:14.159192 6320 ts_tablet_manager.cc:1428] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:40:14.159489 6041 heartbeater.cc:503] Master 127.1.49.62:45389 was elected leader, sending a full tablet report...
I20260926 13:40:14.159704 6303 heartbeater.cc:503] Master 127.1.49.62:45389 was elected leader, sending a full tablet report...
I20260926 13:40:14.159682 6321 consensus_queue.cc:260] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e012afdd5c40188a4a850baf97c5bb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 46673 } } peers { permanent_uuid: "c00031f4b21e419887ca18bce579052a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35929 } } peers { permanent_uuid: "97d79666974c4fb7a7e09627548f6a27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 34257 } }
I20260926 13:40:14.160642 6172 heartbeater.cc:503] Master 127.1.49.62:45389 was elected leader, sending a full tablet report...
I20260926 13:40:14.160730 6321 ts_tablet_manager.cc:1428] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
W20260926 13:40:14.195515 6042 tablet.cc:2369] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:40:14.339946 6304 tablet.cc:2369] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:40:14.381165 6326 raft_consensus.cc:482] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:40:14.381443 6326 raft_consensus.cc:504] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e012afdd5c40188a4a850baf97c5bb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 46673 } } peers { permanent_uuid: "c00031f4b21e419887ca18bce579052a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35929 } } peers { permanent_uuid: "97d79666974c4fb7a7e09627548f6a27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 34257 } }
I20260926 13:40:14.382733 6326 leader_election.cc:288] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06e012afdd5c40188a4a850baf97c5bb (127.1.49.1:46673), c00031f4b21e419887ca18bce579052a (127.1.49.2:35929)
I20260926 13:40:14.384935 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.383290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:40711 (local address 127.1.49.2:35929)
0926 13:40:14.383432 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:14.383435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:14.383448 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:40:14.383486 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:14.383489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:14.383517 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:14.383630 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:14.384469 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:14.384472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:14.384474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:14.384485 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:14.384506 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:14.384514 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:14.384672 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:14.384707 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 13:40:14.384753 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 13:40:14.384948 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.383021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:35929 (local address 127.1.49.3:40711)
0926 13:40:14.383151 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:14.383170 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:14.383263 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:14.383688 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:14.383694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:14.383717 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:14.384245 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:40:14.384246 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:14.384257 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:40:14.384313 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:14.384642 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:14.384645 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:14.384647 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:40:14.384710 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:40:14.384743 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 13:40:14.385107 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.383110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:33015 (local address 127.1.49.1:46673)
0926 13:40:14.383230 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:14.383235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:14.383603 (+ 368us) server_negotiation.cc:407] Connection header received
0926 13:40:14.383708 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:14.383712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:14.383756 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:14.383869 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:14.384671 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:14.384675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:14.384678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:14.384693 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:14.384717 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:14.384728 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:14.384811 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:14.384935 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 13:40:14.384983 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:40:14.385454 6329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.383066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:46673 (local address 127.1.49.3:33015)
0926 13:40:14.383464 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:14.383476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:14.383569 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:14.383894 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:14.383897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:14.383916 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:14.384430 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:40:14.384435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:14.384453 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:40:14.384483 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:14.384817 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:14.384820 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:14.384824 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:40:14.384909 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:40:14.384941 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":134,"threads_started":1}
I20260926 13:40:14.385811 6126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323d1ddcb52a4af8bb2b08a46f040ef6" candidate_uuid: "97d79666974c4fb7a7e09627548f6a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c00031f4b21e419887ca18bce579052a" is_pre_election: true
I20260926 13:40:14.386201 6126 raft_consensus.cc:2454] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97d79666974c4fb7a7e09627548f6a27 in term 0.
I20260926 13:40:14.386199 5995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323d1ddcb52a4af8bb2b08a46f040ef6" candidate_uuid: "97d79666974c4fb7a7e09627548f6a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e012afdd5c40188a4a850baf97c5bb" is_pre_election: true
I20260926 13:40:14.386456 5995 raft_consensus.cc:2454] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97d79666974c4fb7a7e09627548f6a27 in term 0.
I20260926 13:40:14.386884 6193 leader_election.cc:302] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97d79666974c4fb7a7e09627548f6a27, c00031f4b21e419887ca18bce579052a; no voters:
I20260926 13:40:14.387284 6326 raft_consensus.cc:2781] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:40:14.387400 6326 raft_consensus.cc:482] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:40:14.387532 6326 raft_consensus.cc:3037] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:14.388783 6326 raft_consensus.cc:504] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e012afdd5c40188a4a850baf97c5bb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 46673 } } peers { permanent_uuid: "c00031f4b21e419887ca18bce579052a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35929 } } peers { permanent_uuid: "97d79666974c4fb7a7e09627548f6a27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 34257 } }
I20260926 13:40:14.389207 6326 leader_election.cc:288] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [CANDIDATE]: Term 1 election: Requested vote from peers 06e012afdd5c40188a4a850baf97c5bb (127.1.49.1:46673), c00031f4b21e419887ca18bce579052a (127.1.49.2:35929)
I20260926 13:40:14.389690 5995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323d1ddcb52a4af8bb2b08a46f040ef6" candidate_uuid: "97d79666974c4fb7a7e09627548f6a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e012afdd5c40188a4a850baf97c5bb"
I20260926 13:40:14.389689 6126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323d1ddcb52a4af8bb2b08a46f040ef6" candidate_uuid: "97d79666974c4fb7a7e09627548f6a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c00031f4b21e419887ca18bce579052a"
I20260926 13:40:14.389842 5995 raft_consensus.cc:3037] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:14.389842 6126 raft_consensus.cc:3037] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:14.391260 6126 raft_consensus.cc:2454] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97d79666974c4fb7a7e09627548f6a27 in term 1.
I20260926 13:40:14.391260 5995 raft_consensus.cc:2454] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97d79666974c4fb7a7e09627548f6a27 in term 1.
W20260926 13:40:14.391532 6173 tablet.cc:2369] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:40:14.391723 6192 leader_election.cc:302] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e012afdd5c40188a4a850baf97c5bb, 97d79666974c4fb7a7e09627548f6a27; no voters:
I20260926 13:40:14.392055 6326 raft_consensus.cc:2781] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:14.392326 6326 raft_consensus.cc:686] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [term 1 LEADER]: Becoming Leader. State: Replica: 97d79666974c4fb7a7e09627548f6a27, State: Running, Role: LEADER
I20260926 13:40:14.392704 6326 consensus_queue.cc:237] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e012afdd5c40188a4a850baf97c5bb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 46673 } } peers { permanent_uuid: "c00031f4b21e419887ca18bce579052a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35929 } } peers { permanent_uuid: "97d79666974c4fb7a7e09627548f6a27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 34257 } }
I20260926 13:40:14.396214 5852 catalog_manager.cc:5495] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97d79666974c4fb7a7e09627548f6a27 (127.1.49.3). New cstate: current_term: 1 leader_uuid: "97d79666974c4fb7a7e09627548f6a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e012afdd5c40188a4a850baf97c5bb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 46673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c00031f4b21e419887ca18bce579052a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97d79666974c4fb7a7e09627548f6a27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 34257 } health_report { overall_health: HEALTHY } } }
I20260926 13:40:14.432192 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.430293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.1.49.3:34257)
0926 13:40:14.430530 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:14.430534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:14.430548 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:40:14.430626 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:14.430629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:14.430686 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:14.430757 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:14.431609 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:14.431613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:14.431615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:14.431628 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:14.431648 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:14.431657 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:14.431748 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:14.431913 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 13:40:14.431994 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260926 13:40:14.439553 6126 raft_consensus.cc:1261] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a [term 1 FOLLOWER]: Refusing update from remote peer 97d79666974c4fb7a7e09627548f6a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:14.439702 5995 raft_consensus.cc:1261] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb [term 1 FOLLOWER]: Refusing update from remote peer 97d79666974c4fb7a7e09627548f6a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:14.440634 6326 consensus_queue.cc:1035] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c00031f4b21e419887ca18bce579052a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:14.440966 6330 consensus_queue.cc:1035] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e012afdd5c40188a4a850baf97c5bb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 46673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:14.450563 6335 mvcc.cc:205] Tried to move back new op lower bound from 7333601339134779392 to 7333601338955362304. Current Snapshot: MvccSnapshot[applied={T|T < 7333601339134779392}]
I20260926 13:40:14.450958 6333 mvcc.cc:205] Tried to move back new op lower bound from 7333601339134779392 to 7333601338955362304. Current Snapshot: MvccSnapshot[applied={T|T < 7333601339134779392}]
I20260926 13:40:14.451781 6334 mvcc.cc:205] Tried to move back new op lower bound from 7333601339134779392 to 7333601338955362304. Current Snapshot: MvccSnapshot[applied={T|T < 7333601339134779392}]
I20260926 13:40:14.462704 5850 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:14.462961 5850 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:14.466342 5850 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48b90f1e9b0b488ba3fb6e5e6e84a27f: Sending DeleteTablet for 3 replicas of tablet 323d1ddcb52a4af8bb2b08a46f040ef6
I20260926 13:40:14.467489 5975 tablet_service.cc:1514] Processing DeleteTablet for tablet 323d1ddcb52a4af8bb2b08a46f040ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:14 UTC) from {username='slave'} at 127.0.0.1:34112
I20260926 13:40:14.467519 6237 tablet_service.cc:1514] Processing DeleteTablet for tablet 323d1ddcb52a4af8bb2b08a46f040ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:14 UTC) from {username='slave'} at 127.0.0.1:58158
I20260926 13:40:14.467624 6106 tablet_service.cc:1514] Processing DeleteTablet for tablet 323d1ddcb52a4af8bb2b08a46f040ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:14 UTC) from {username='slave'} at 127.0.0.1:37502
I20260926 13:40:14.468196 6342 tablet_replica.cc:330] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb: stopping tablet replica
I20260926 13:40:14.468320 6344 tablet_replica.cc:330] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a: stopping tablet replica
I20260926 13:40:14.468539 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 5913
I20260926 13:40:14.468636 6342 raft_consensus.cc:2229] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 06e012afdd5c40188a4a850baf97c5bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:14.469537 6344 raft_consensus.cc:2229] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:14.469897 6343 tablet_replica.cc:330] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27: stopping tablet replica
I20260926 13:40:14.469933 6344 raft_consensus.cc:2258] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:14.470257 6343 raft_consensus.cc:2229] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:40:14.470701 6343 raft_consensus.cc:2258] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:14.471715 6344 ts_tablet_manager.cc:1905] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:14.475550 6344 ts_tablet_manager.cc:1918] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:14.475693 6344 log.cc:1196] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/323d1ddcb52a4af8bb2b08a46f040ef6
I20260926 13:40:14.476096 6344 ts_tablet_manager.cc:1939] T 323d1ddcb52a4af8bb2b08a46f040ef6 P c00031f4b21e419887ca18bce579052a: Deleting consensus metadata
W20260926 13:40:14.477818 5839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:46673 (error 108)
I20260926 13:40:14.478434 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 6044
I20260926 13:40:14.478434 5840 catalog_manager.cc:4841] TS c00031f4b21e419887ca18bce579052a (127.1.49.2:35929): tablet 323d1ddcb52a4af8bb2b08a46f040ef6 (table test-table [id=4bb65827edaf44c9816be3aba0f60d43]) successfully deleted
I20260926 13:40:14.479580 6317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.479207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:46673 (local address 127.0.0.1:34126)
0926 13:40:14.479407 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:14.479484 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260926 13:40:14.479766 5839 catalog_manager.cc:4584] TS 06e012afdd5c40188a4a850baf97c5bb (127.1.49.1:46673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 323d1ddcb52a4af8bb2b08a46f040ef6: Network error: Client connection negotiation failed: client connection to 127.1.49.1:46673: connect: Connection refused (error 111)
I20260926 13:40:14.480157 6343 ts_tablet_manager.cc:1905] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:14.484411 6343 ts_tablet_manager.cc:1918] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:14.484563 6343 log.cc:1196] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/323d1ddcb52a4af8bb2b08a46f040ef6
I20260926 13:40:14.484958 6343 ts_tablet_manager.cc:1939] T 323d1ddcb52a4af8bb2b08a46f040ef6 P 97d79666974c4fb7a7e09627548f6a27: Deleting consensus metadata
I20260926 13:40:14.486685 5840 catalog_manager.cc:4841] TS 97d79666974c4fb7a7e09627548f6a27 (127.1.49.3:34257): tablet 323d1ddcb52a4af8bb2b08a46f040ef6 (table test-table [id=4bb65827edaf44c9816be3aba0f60d43]) successfully deleted
I20260926 13:40:14.488168 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 6175
I20260926 13:40:14.493930 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 5822
2026-09-26T13:40:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1270 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 13:40:14.517684 1220 mini_oidc.cc:152] Starting JWKS server
I20260926 13:40:14.519436 1220 webserver.cc:460] Webserver started at https://127.0.0.1:36715/ using document root <none> and password file <none>
I20260926 13:40:14.519656 1220 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 13:40:14.520046 1220 webserver.cc:460] Webserver started at http://127.0.0.1:40803/ using document root <none> and password file <none>
2026-09-26T13:40:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:14Z Disabled control of system clock
I20260926 13:40:14.532174 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:46807
--webserver_interface=127.1.49.62
--webserver_port=0
--builtin_ntp_servers=127.1.49.20:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:46807
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36715/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/testchainca.cert with env {}
W20260926 13:40:14.662247 6351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:14.662556 6351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:14.662599 6351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:14.666929 6351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:14.667019 6351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:14.667035 6351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:14.667069 6351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:14.667086 6351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:40:14.667104 6351 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:40:14.667117 6351 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36715/jwks.json
I20260926 13:40:14.673169 6351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:46807
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:46807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36715/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:14.674516 6351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:14.675653 6351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:14.685101 6356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:14.685146 6357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:14.685125 6359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:14.685658 6351 server_base.cc:1034] running on GCE node
I20260926 13:40:14.686326 6351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:14.687724 6351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:14.688944 6351 hybrid_clock.cc:648] HybridClock initialized: now 1790430014688923 us; error 39 us; skew 500 ppm
I20260926 13:40:14.691501 6351 webserver.cc:460] Webserver started at http://127.1.49.62:36983/ using document root <none> and password file <none>
I20260926 13:40:14.692256 6351 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:14.692333 6351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:14.692581 6351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:14.694949 6351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "9f592ac4aed94711a559e35a74d1b6db"
format_stamp: "Formatted at 2026-09-26 13:40:14 on dist-test-slave-1wns"
server_key: "b714e7261d5cb983daa08ebaa5f6b3b2"
server_key_iv: "68c69d7eb07c4195e1c237b77b58ca38"
server_key_version: "encryptionkey@0"
I20260926 13:40:14.695639 6351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "9f592ac4aed94711a559e35a74d1b6db"
format_stamp: "Formatted at 2026-09-26 13:40:14 on dist-test-slave-1wns"
server_key: "b714e7261d5cb983daa08ebaa5f6b3b2"
server_key_iv: "68c69d7eb07c4195e1c237b77b58ca38"
server_key_version: "encryptionkey@0"
I20260926 13:40:14.700135 6351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:40:14.703020 6365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:14.704573 6351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:14.704708 6351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "9f592ac4aed94711a559e35a74d1b6db"
format_stamp: "Formatted at 2026-09-26 13:40:14 on dist-test-slave-1wns"
server_key: "b714e7261d5cb983daa08ebaa5f6b3b2"
server_key_iv: "68c69d7eb07c4195e1c237b77b58ca38"
server_key_version: "encryptionkey@0"
I20260926 13:40:14.704878 6351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:14.719964 6351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:14.720772 6351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:14.720975 6351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:14.728752 6351 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:46807
I20260926 13:40:14.728757 6417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:46807 every 8 connection(s)
I20260926 13:40:14.729959 6351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:14.730350 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 6351
I20260926 13:40:14.730482 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:14.730748 1220 external_mini_cluster.cc:1428] Setting key 9d3ecd0c377693a9f08aa4908fdc9998
I20260926 13:40:14.734120 6418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:14.740881 6418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db: Bootstrap starting.
I20260926 13:40:14.743170 6418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:14.743928 6418 log.cc:824] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:14.746461 6418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db: No bootstrap required, opened a new log
I20260926 13:40:14.746812 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.732284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.1.49.62:46807)
0926 13:40:14.733124 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:14.733145 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:14.733191 (+ 46us) server_negotiation.cc:407] Connection header received
0926 13:40:14.734095 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:14.734120 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:14.734656 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:14.735044 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:14.735510 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:14.737004 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:14.737947 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:14.738659 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:14.738814 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:14.744877 (+ 6063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:14.744883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:14.744898 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:14.745324 (+ 426us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:14.745382 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:14.745405 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:14.745542 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:14.745714 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 13:40:14.746005 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":119,"threads_started":1}
I20260926 13:40:14.749714 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:40:14.751001 6418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f592ac4aed94711a559e35a74d1b6db" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46807 } }
I20260926 13:40:14.751281 6418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:14.751368 6418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f592ac4aed94711a559e35a74d1b6db, State: Initialized, Role: FOLLOWER
I20260926 13:40:14.751977 6418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f592ac4aed94711a559e35a74d1b6db" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46807 } }
I20260926 13:40:14.752125 6418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:14.752214 6418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:14.752334 6418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:14.753391 6418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f592ac4aed94711a559e35a74d1b6db" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46807 } }
I20260926 13:40:14.753818 6418 leader_election.cc:302] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f592ac4aed94711a559e35a74d1b6db; no voters:
I20260926 13:40:14.754182 6418 leader_election.cc:288] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:14.754364 6423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:14.754642 6423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [term 1 LEADER]: Becoming Leader. State: Replica: 9f592ac4aed94711a559e35a74d1b6db, State: Running, Role: LEADER
I20260926 13:40:14.755023 6423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f592ac4aed94711a559e35a74d1b6db" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46807 } }
I20260926 13:40:14.755636 6418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:14.756317 6423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f592ac4aed94711a559e35a74d1b6db. Latest consensus state: current_term: 1 leader_uuid: "9f592ac4aed94711a559e35a74d1b6db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f592ac4aed94711a559e35a74d1b6db" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46807 } } }
I20260926 13:40:14.756433 6423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:14.756837 6425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f592ac4aed94711a559e35a74d1b6db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f592ac4aed94711a559e35a74d1b6db" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46807 } } }
I20260926 13:40:14.756949 6425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:14.757020 6428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:14.760785 6428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:14.767660 6428 catalog_manager.cc:1347] Generated new cluster ID: 87225613f5d3429fa3a70d9ffa51df65
I20260926 13:40:14.767803 6428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:14.785708 6428 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:14.786828 6428 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:14.800959 6428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f592ac4aed94711a559e35a74d1b6db: Generated new TSK 0
I20260926 13:40:14.801882 6428 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 13:40:14.833297 6448 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 13:40:14.833525 6448 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 13:40:14.834053 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:14.814322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.1.49.62:46807)
0926 13:40:14.814502 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:14.814513 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:14.814737 (+ 224us) server_negotiation.cc:407] Connection header received
0926 13:40:14.814966 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:14.814969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:14.815039 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:14.815171 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:40:14.815490 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:14.816373 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:14.817484 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:14.817831 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:14.817950 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:14.818587 (+ 637us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:40:14.833615 (+ 15028us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 13:40:14.833908 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53094: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 13:40:14.834123 6421 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53094: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 13:40:14.835348 6447 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:14.814165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46807 (local address 127.0.0.1:53094)
0926 13:40:14.814628 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:14.814643 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:14.814786 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:14.815232 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:14.815235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:14.815244 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:40:14.815353 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:14.815359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:14.816588 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:14.816592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:14.817350 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:14.817359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:14.817971 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:14.817999 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:14.818080 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:14.818137 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:40:14.833861 (+ 15724us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 13:40:14.834093 (+ 232us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.49.62:46807: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":149,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.49.62:46807: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 13:40:14.836254 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 6351
2026-09-26T13:40:14Z chronyd exiting
I20260926 13:40:14.873478 1220 test_util.cc:175] -----------------------------------------------
I20260926 13:40:14.873581 1220 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790429983839632-1220-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (357 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 13:40:14.874769 1220 mini_oidc.cc:152] Starting JWKS server
I20260926 13:40:14.875770 1220 webserver.cc:460] Webserver started at https://127.0.0.1:45783/ using document root <none> and password file <none>
I20260926 13:40:14.875967 1220 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 13:40:14.876276 1220 webserver.cc:460] Webserver started at http://127.0.0.1:37249/ using document root <none> and password file <none>
2026-09-26T13:40:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:14Z Disabled control of system clock
I20260926 13:40:14.886698 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:42425
--webserver_interface=127.1.49.62
--webserver_port=0
--builtin_ntp_servers=127.1.49.20:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:42425
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45783/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/testchainca.cert with env {}
W20260926 13:40:15.011518 6454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:15.011986 6454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:15.012043 6454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:15.016594 6454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:15.016678 6454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:15.016695 6454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:15.016712 6454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:15.016729 6454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:40:15.016746 6454 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:40:15.016759 6454 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45783/jwks.json
I20260926 13:40:15.022914 6454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:42425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:42425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45783/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:15.024166 6454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:15.025305 6454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:15.035926 6462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:15.035897 6460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:15.035867 6459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:15.036486 6454 server_base.cc:1034] running on GCE node
I20260926 13:40:15.037249 6454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:15.038514 6454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:15.039708 6454 hybrid_clock.cc:648] HybridClock initialized: now 1790430015039666 us; error 61 us; skew 500 ppm
I20260926 13:40:15.041971 6454 webserver.cc:460] Webserver started at http://127.1.49.62:46817/ using document root <none> and password file <none>
I20260926 13:40:15.042663 6454 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:15.042771 6454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:15.043072 6454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:15.045538 6454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "bf3bf4a855d04b76a5a09e9e8c01c7d7"
format_stamp: "Formatted at 2026-09-26 13:40:15 on dist-test-slave-1wns"
server_key: "36274de7f16bd6be1dbd5cfea8107841"
server_key_iv: "ba1170830a9bb58bafa6b81eb04f9e5d"
server_key_version: "encryptionkey@0"
I20260926 13:40:15.046245 6454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "bf3bf4a855d04b76a5a09e9e8c01c7d7"
format_stamp: "Formatted at 2026-09-26 13:40:15 on dist-test-slave-1wns"
server_key: "36274de7f16bd6be1dbd5cfea8107841"
server_key_iv: "ba1170830a9bb58bafa6b81eb04f9e5d"
server_key_version: "encryptionkey@0"
I20260926 13:40:15.050740 6454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 13:40:15.054033 6468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:15.055572 6454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:40:15.055756 6454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "bf3bf4a855d04b76a5a09e9e8c01c7d7"
format_stamp: "Formatted at 2026-09-26 13:40:15 on dist-test-slave-1wns"
server_key: "36274de7f16bd6be1dbd5cfea8107841"
server_key_iv: "ba1170830a9bb58bafa6b81eb04f9e5d"
server_key_version: "encryptionkey@0"
I20260926 13:40:15.055971 6454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:15.073274 6454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:15.074123 6454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:15.074353 6454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:15.082100 6454 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:42425
I20260926 13:40:15.082171 6520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:42425 every 8 connection(s)
I20260926 13:40:15.083206 6454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:15.083979 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 6454
I20260926 13:40:15.084120 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:15.084414 1220 external_mini_cluster.cc:1428] Setting key 1c0d67cddb41fc94379776d4823a526b
I20260926 13:40:15.086994 6521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:15.093003 6521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7: Bootstrap starting.
I20260926 13:40:15.094077 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:15.085793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34662 (local address 127.1.49.62:42425)
0926 13:40:15.086510 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:15.086533 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:15.086581 (+ 48us) server_negotiation.cc:407] Connection header received
0926 13:40:15.087320 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:15.087345 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:15.087758 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:15.088179 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:15.088624 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.089611 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.090518 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.091020 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.091222 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:15.092185 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:15.092191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:15.092205 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:15.092627 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:15.092689 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:15.092709 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:15.092822 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:15.092961 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 13:40:15.093290 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":145,"threads_started":1}
I20260926 13:40:15.095623 6521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:15.096664 6521 log.cc:824] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:15.097550 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:40:15.099479 6521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7: No bootstrap required, opened a new log
I20260926 13:40:15.103696 6521 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3bf4a855d04b76a5a09e9e8c01c7d7" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42425 } }
I20260926 13:40:15.104107 6521 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:15.104189 6521 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3bf4a855d04b76a5a09e9e8c01c7d7, State: Initialized, Role: FOLLOWER
I20260926 13:40:15.104761 6521 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3bf4a855d04b76a5a09e9e8c01c7d7" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42425 } }
I20260926 13:40:15.104948 6521 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:15.105036 6521 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:15.105188 6521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:15.106324 6521 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3bf4a855d04b76a5a09e9e8c01c7d7" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42425 } }
I20260926 13:40:15.106693 6521 leader_election.cc:302] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf3bf4a855d04b76a5a09e9e8c01c7d7; no voters:
I20260926 13:40:15.107008 6521 leader_election.cc:288] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:15.107156 6526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:15.107409 6526 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [term 1 LEADER]: Becoming Leader. State: Replica: bf3bf4a855d04b76a5a09e9e8c01c7d7, State: Running, Role: LEADER
I20260926 13:40:15.107764 6526 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3bf4a855d04b76a5a09e9e8c01c7d7" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42425 } }
I20260926 13:40:15.108604 6521 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:15.109891 6527 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf3bf4a855d04b76a5a09e9e8c01c7d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3bf4a855d04b76a5a09e9e8c01c7d7" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42425 } } }
I20260926 13:40:15.110088 6527 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:15.110467 6528 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf3bf4a855d04b76a5a09e9e8c01c7d7. Latest consensus state: current_term: 1 leader_uuid: "bf3bf4a855d04b76a5a09e9e8c01c7d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3bf4a855d04b76a5a09e9e8c01c7d7" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42425 } } }
I20260926 13:40:15.110600 6528 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:15.110548 6535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:15.113863 6535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:15.121040 6535 catalog_manager.cc:1347] Generated new cluster ID: 4dd4d64fd4834e3289b07c53ccd6a347
I20260926 13:40:15.121145 6535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:15.151737 6535 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:15.153169 6535 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:15.162626 6535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf3bf4a855d04b76a5a09e9e8c01c7d7: Generated new TSK 0
I20260926 13:40:15.163529 6535 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 13:40:15.197582 6551 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 13:40:15.197700 6551 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 13:40:15.198553 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:15.172732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.1.49.62:42425)
0926 13:40:15.172901 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:15.172906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:15.173094 (+ 188us) server_negotiation.cc:407] Connection header received
0926 13:40:15.173252 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:15.173255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:15.173311 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:15.173440 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:40:15.173706 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.174350 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.175671 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.175995 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.176148 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:15.176532 (+ 384us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:40:15.198031 (+ 21499us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 13:40:15.198385 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 13:40:15.198640 6524 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 13:40:15.198690 6550 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:15.172656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:42425 (local address 127.0.0.1:34668)
0926 13:40:15.172981 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:15.172997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:15.173127 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:15.173449 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:15.173452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:15.173460 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:40:15.173578 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:15.173584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.174474 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.174478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:15.175504 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:15.175518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.176138 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.176171 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:15.176281 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:15.176311 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:40:15.198289 (+ 21978us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 13:40:15.198477 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.49.62:42425: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":128,"threads_started":1}
I20260926 13:40:15.199482 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 6454
2026-09-26T13:40:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (364 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 13:40:15.239691 1220 mini_oidc.cc:152] Starting JWKS server
I20260926 13:40:15.240736 1220 webserver.cc:460] Webserver started at https://127.0.0.1:44357/ using document root <none> and password file <none>
I20260926 13:40:15.240878 1220 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 13:40:15.241173 1220 webserver.cc:460] Webserver started at http://127.0.0.1:36593/ using document root <none> and password file <none>
2026-09-26T13:40:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:15Z Disabled control of system clock
I20260926 13:40:15.251662 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:34037
--webserver_interface=127.1.49.62
--webserver_port=0
--builtin_ntp_servers=127.1.49.20:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:34037
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44357/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 13:40:15.390575 6557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:15.390931 6557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:15.390978 6557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:15.395743 6557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:15.395859 6557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:15.395876 6557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:15.395896 6557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:15.395912 6557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:40:15.395955 6557 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:40:15.395982 6557 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44357/jwks.json
I20260926 13:40:15.401190 6557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:34037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:34037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44357/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:15.402446 6557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:15.403990 6557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:15.413161 6563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:15.413161 6565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:15.413213 6562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:15.414083 6557 server_base.cc:1034] running on GCE node
I20260926 13:40:15.414707 6557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:15.416054 6557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:15.417251 6557 hybrid_clock.cc:648] HybridClock initialized: now 1790430015417230 us; error 36 us; skew 500 ppm
I20260926 13:40:15.419384 6557 webserver.cc:460] Webserver started at http://127.1.49.62:35069/ using document root <none> and password file <none>
I20260926 13:40:15.420078 6557 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:15.420145 6557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:15.420362 6557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:15.422590 6557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "3fc177cf5783478dbfc05720b02c7958"
format_stamp: "Formatted at 2026-09-26 13:40:15 on dist-test-slave-1wns"
server_key: "70c5cc3063edfddd9fa1247f8ce34f12"
server_key_iv: "cdb5f9501eac9635d1a8338e522377ee"
server_key_version: "encryptionkey@0"
I20260926 13:40:15.423199 6557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "3fc177cf5783478dbfc05720b02c7958"
format_stamp: "Formatted at 2026-09-26 13:40:15 on dist-test-slave-1wns"
server_key: "70c5cc3063edfddd9fa1247f8ce34f12"
server_key_iv: "cdb5f9501eac9635d1a8338e522377ee"
server_key_version: "encryptionkey@0"
I20260926 13:40:15.427315 6557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 13:40:15.430320 6571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:15.432156 6557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:15.432333 6557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "3fc177cf5783478dbfc05720b02c7958"
format_stamp: "Formatted at 2026-09-26 13:40:15 on dist-test-slave-1wns"
server_key: "70c5cc3063edfddd9fa1247f8ce34f12"
server_key_iv: "cdb5f9501eac9635d1a8338e522377ee"
server_key_version: "encryptionkey@0"
I20260926 13:40:15.432482 6557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:15.486579 6557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:15.487402 6557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:15.487625 6557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:15.495007 6557 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:34037
I20260926 13:40:15.495074 6623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:34037 every 8 connection(s)
I20260926 13:40:15.496109 6557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:15.500078 6624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:15.500123 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 6557
I20260926 13:40:15.500254 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:15.500558 1220 external_mini_cluster.cc:1428] Setting key 5aefe61a49c7d7f7b58b0e55a6c96538
I20260926 13:40:15.508111 6624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958: Bootstrap starting.
I20260926 13:40:15.510785 6624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:15.511200 6627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:15.502142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.1.49.62:34037)
0926 13:40:15.502822 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:15.502841 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:15.502879 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:40:15.503797 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:15.503836 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:15.504316 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:15.504624 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:15.504997 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.506050 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.507379 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.508115 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.508311 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:15.509117 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:15.509123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:15.509136 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:15.509641 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:15.509703 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:15.509724 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:15.509803 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:15.509979 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 13:40:15.510296 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":157,"threads_started":1}
I20260926 13:40:15.511647 6624 log.cc:824] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:15.513841 6624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958: No bootstrap required, opened a new log
I20260926 13:40:15.514489 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:40:15.517863 6624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc177cf5783478dbfc05720b02c7958" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34037 } }
I20260926 13:40:15.518123 6624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:15.518158 6624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc177cf5783478dbfc05720b02c7958, State: Initialized, Role: FOLLOWER
I20260926 13:40:15.518548 6624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc177cf5783478dbfc05720b02c7958" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34037 } }
I20260926 13:40:15.518661 6624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:15.518702 6624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:15.518766 6624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:15.519721 6624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc177cf5783478dbfc05720b02c7958" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34037 } }
I20260926 13:40:15.520129 6624 leader_election.cc:302] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fc177cf5783478dbfc05720b02c7958; no voters:
I20260926 13:40:15.520489 6624 leader_election.cc:288] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:15.520702 6629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:15.521042 6629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [term 1 LEADER]: Becoming Leader. State: Replica: 3fc177cf5783478dbfc05720b02c7958, State: Running, Role: LEADER
I20260926 13:40:15.521447 6629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc177cf5783478dbfc05720b02c7958" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34037 } }
I20260926 13:40:15.522066 6624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:15.523491 6630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fc177cf5783478dbfc05720b02c7958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc177cf5783478dbfc05720b02c7958" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34037 } } }
I20260926 13:40:15.523438 6631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fc177cf5783478dbfc05720b02c7958. Latest consensus state: current_term: 1 leader_uuid: "3fc177cf5783478dbfc05720b02c7958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc177cf5783478dbfc05720b02c7958" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34037 } } }
I20260926 13:40:15.523607 6630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:15.523607 6631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:15.524135 6639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:15.527994 6639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:15.534375 6639 catalog_manager.cc:1347] Generated new cluster ID: a103c21ebe75429ab75370890a7059d0
I20260926 13:40:15.534478 6639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:15.544241 6639 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:15.545588 6639 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:15.555913 6639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fc177cf5783478dbfc05720b02c7958: Generated new TSK 0
I20260926 13:40:15.556864 6639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:40:15.575541 6627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:15.569880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.1.49.62:34037)
0926 13:40:15.570041 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:15.570048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:15.570278 (+ 230us) server_negotiation.cc:407] Connection header received
0926 13:40:15.570455 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:15.570458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:15.570522 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:15.570657 (+ 135us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:40:15.570908 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.571769 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.572757 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.573110 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.573214 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:15.573552 (+ 338us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:40:15.575102 (+ 1550us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44357/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 13:40:15.575363 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48802: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44357/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 13:40:15.575627 6627 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48802: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44357/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 13:40:15.577273 6653 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:15.569748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:34037 (local address 127.0.0.1:48802)
0926 13:40:15.570174 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:15.570189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:15.570303 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:15.570671 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:15.570674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:15.570682 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:40:15.570796 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:15.570801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.571930 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.571934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:15.572637 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:15.572645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.573229 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.573252 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:15.573327 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:15.573354 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:40:15.575286 (+ 1932us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44357/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 13:40:15.577096 (+ 1810us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.49.62:34037: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44357/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":1503,"thread_start_us":238,"threads_started":1}
I20260926 13:40:15.577695 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 6557
E20260926 13:40:15.586505 6654 webserver.cc:533] Webserver: sslize failed:
E20260926 13:40:15.586603 6654 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T13:40:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (377 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T13:40:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:15Z Disabled control of system clock
I20260926 13:40:15.628943 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:36121
--webserver_interface=127.1.49.62
--webserver_port=0
--builtin_ntp_servers=127.1.49.20:43595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:36121
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/testchainca.cert with env {}
W20260926 13:40:15.751530 6658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:15.751869 6658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:15.751916 6658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:15.756074 6658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:15.756138 6658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:15.756155 6658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:15.756172 6658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:15.756188 6658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:40:15.756206 6658 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:40:15.756219 6658 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 13:40:15.761871 6658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:43595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:36121
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:36121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:15.763000 6658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:15.764158 6658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:15.773455 6663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:15.773548 6666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:15.773600 6664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:15.773963 6658 server_base.cc:1034] running on GCE node
I20260926 13:40:15.774580 6658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:15.775908 6658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:15.777081 6658 hybrid_clock.cc:648] HybridClock initialized: now 1790430015777066 us; error 31 us; skew 500 ppm
I20260926 13:40:15.779069 6658 webserver.cc:460] Webserver started at http://127.1.49.62:36099/ using document root <none> and password file <none>
I20260926 13:40:15.779726 6658 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:15.779793 6658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:15.780077 6658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:15.782187 6658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "a298b3d58668474e8b652d85eb753946"
format_stamp: "Formatted at 2026-09-26 13:40:15 on dist-test-slave-1wns"
server_key: "519f6673ac79f77bca1a9e958974b4f4"
server_key_iv: "b4e72c2e88dc556ccab44660ec290397"
server_key_version: "encryptionkey@0"
I20260926 13:40:15.782757 6658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "a298b3d58668474e8b652d85eb753946"
format_stamp: "Formatted at 2026-09-26 13:40:15 on dist-test-slave-1wns"
server_key: "519f6673ac79f77bca1a9e958974b4f4"
server_key_iv: "b4e72c2e88dc556ccab44660ec290397"
server_key_version: "encryptionkey@0"
I20260926 13:40:15.786741 6658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 13:40:15.790025 6672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:15.791528 6658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:40:15.791668 6658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "a298b3d58668474e8b652d85eb753946"
format_stamp: "Formatted at 2026-09-26 13:40:15 on dist-test-slave-1wns"
server_key: "519f6673ac79f77bca1a9e958974b4f4"
server_key_iv: "b4e72c2e88dc556ccab44660ec290397"
server_key_version: "encryptionkey@0"
I20260926 13:40:15.791791 6658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:15.808286 6658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:15.809054 6658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:15.809222 6658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:15.816870 6658 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:36121
I20260926 13:40:15.816877 6724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:36121 every 8 connection(s)
I20260926 13:40:15.817966 6658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:15.821887 6725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:15.826462 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 6658
I20260926 13:40:15.826601 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:15.826887 1220 external_mini_cluster.cc:1428] Setting key 7bb54c598653dd51e030b4bfa35e9ede
I20260926 13:40:15.829070 6725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946: Bootstrap starting.
I20260926 13:40:15.832403 6725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:15.833516 6725 log.cc:824] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:15.836452 6725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946: No bootstrap required, opened a new log
I20260926 13:40:15.839452 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:15.828579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.1.49.62:36121)
0926 13:40:15.829368 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:15.829390 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:15.829440 (+ 50us) server_negotiation.cc:407] Connection header received
0926 13:40:15.830427 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:15.830458 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:15.831175 (+ 717us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:15.831631 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:15.832289 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.834065 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.835228 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.836003 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.836182 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:15.837142 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:15.837148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:15.837167 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:15.837704 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:15.837784 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:15.837809 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:15.837922 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:15.838112 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 13:40:15.838422 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":118,"threads_started":1}
I20260926 13:40:15.841306 6725 raft_consensus.cc:348] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a298b3d58668474e8b652d85eb753946" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36121 } }
I20260926 13:40:15.841614 6725 raft_consensus.cc:374] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:15.841699 6725 raft_consensus.cc:729] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a298b3d58668474e8b652d85eb753946, State: Initialized, Role: FOLLOWER
I20260926 13:40:15.842151 6725 consensus_queue.cc:260] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a298b3d58668474e8b652d85eb753946" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36121 } }
I20260926 13:40:15.842334 6725 raft_consensus.cc:388] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:15.842428 6725 raft_consensus.cc:482] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:15.842576 6725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:15.842924 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:40:15.843677 6725 raft_consensus.cc:504] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a298b3d58668474e8b652d85eb753946" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36121 } }
I20260926 13:40:15.844122 6725 leader_election.cc:302] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a298b3d58668474e8b652d85eb753946; no voters:
I20260926 13:40:15.844494 6725 leader_election.cc:288] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:15.844632 6730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:15.844877 6730 raft_consensus.cc:686] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [term 1 LEADER]: Becoming Leader. State: Replica: a298b3d58668474e8b652d85eb753946, State: Running, Role: LEADER
I20260926 13:40:15.845281 6730 consensus_queue.cc:237] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a298b3d58668474e8b652d85eb753946" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36121 } }
I20260926 13:40:15.845817 6725 sys_catalog.cc:564] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:15.847002 6731 sys_catalog.cc:455] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a298b3d58668474e8b652d85eb753946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a298b3d58668474e8b652d85eb753946" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36121 } } }
I20260926 13:40:15.847165 6731 sys_catalog.cc:458] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:15.847469 6732 sys_catalog.cc:455] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a298b3d58668474e8b652d85eb753946. Latest consensus state: current_term: 1 leader_uuid: "a298b3d58668474e8b652d85eb753946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a298b3d58668474e8b652d85eb753946" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36121 } } }
I20260926 13:40:15.847581 6739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:15.847591 6732 sys_catalog.cc:458] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:15.851176 6739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:15.857831 6739 catalog_manager.cc:1347] Generated new cluster ID: e962e216854943a9b5566feed5b3169f
I20260926 13:40:15.857930 6739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:15.891791 6739 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:15.893187 6739 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:15.899526 6739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a298b3d58668474e8b652d85eb753946: Generated new TSK 0
I20260926 13:40:15.900538 6739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:40:15.995003 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:15.914015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.1.49.62:36121)
0926 13:40:15.914169 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:15.914176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:15.914346 (+ 170us) server_negotiation.cc:407] Connection header received
0926 13:40:15.914510 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:15.914514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:15.914577 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:15.914706 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:40:15.914943 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.915839 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.917001 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.917366 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.917472 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:15.917983 (+ 511us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:40:15.994454 (+ 76471us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 13:40:15.994805 (+ 351us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57222: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 13:40:15.995115 6728 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57222: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 13:40:15.995769 6754 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:15.913860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36121 (local address 127.0.0.1:57222)
0926 13:40:15.914247 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:15.914264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:15.914373 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:15.914716 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:15.914718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:15.914726 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:40:15.914830 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:15.914834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.915971 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.915975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:15.916861 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:15.916872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:15.917544 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:15.917571 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:15.917747 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:15.917777 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:40:15.994759 (+ 76982us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 13:40:15.995014 (+ 255us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.49.62:36121: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":147,"threads_started":1}
I20260926 13:40:15.996919 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 6658
2026-09-26T13:40:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (400 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (137 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (138 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:40:16 dist-test-slave-1wns krb5kdc[6766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:40:16 dist-test-slave-1wns krb5kdc[6766](info): set up 2 sockets
Sep 26 13:40:16 dist-test-slave-1wns krb5kdc[6766](info): commencing operation
krb5kdc: starting...
W20260926 13:40:17.408382 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:40:17 dist-test-slave-1wns krb5kdc[6766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:40:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:40:17.540606 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:40993
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:40993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:17.656776 6779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:17.657085 6779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:17.657128 6779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:17.661381 6779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:17.661464 6779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:17.661485 6779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:17.661504 6779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:17.661520 6779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:17.667150 6779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:40993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:40993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:17.668396 6779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:17.669498 6779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:17.679076 6785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:17.679086 6787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:17.679150 6779 server_base.cc:1034] running on GCE node
W20260926 13:40:17.679078 6784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:17.680282 6779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:17.681641 6779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:17.682839 6779 hybrid_clock.cc:648] HybridClock initialized: now 1790430017682799 us; error 52 us; skew 500 ppm
Sep 26 13:40:17 dist-test-slave-1wns krb5kdc[6766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:17.686057 6779 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:40:17.687150 6779 webserver.cc:460] Webserver started at http://127.1.49.62:42497/ using document root <none> and password file <none>
I20260926 13:40:17.687745 6779 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:17.687918 6779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:17.688161 6779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:17.690495 6779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "5bbc914116d2454e980a37a591f37b36"
format_stamp: "Formatted at 2026-09-26 13:40:17 on dist-test-slave-1wns"
server_key: "1031ed915e79e36ba4539c5d2a24cc65"
server_key_iv: "49069dd812464c69cc135fc418d7df90"
server_key_version: "encryptionkey@0"
I20260926 13:40:17.691092 6779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "5bbc914116d2454e980a37a591f37b36"
format_stamp: "Formatted at 2026-09-26 13:40:17 on dist-test-slave-1wns"
server_key: "1031ed915e79e36ba4539c5d2a24cc65"
server_key_iv: "49069dd812464c69cc135fc418d7df90"
server_key_version: "encryptionkey@0"
I20260926 13:40:17.695878 6779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 13:40:17.698819 6794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:17.700605 6779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:17.700809 6779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "5bbc914116d2454e980a37a591f37b36"
format_stamp: "Formatted at 2026-09-26 13:40:17 on dist-test-slave-1wns"
server_key: "1031ed915e79e36ba4539c5d2a24cc65"
server_key_iv: "49069dd812464c69cc135fc418d7df90"
server_key_version: "encryptionkey@0"
I20260926 13:40:17.700965 6779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:17.726943 6779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:17.731346 6779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:17.731637 6779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:17.740916 6779 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:40993
I20260926 13:40:17.740972 6846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:40993 every 8 connection(s)
I20260926 13:40:17.742112 6779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:17.745615 6847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:17.748670 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 6779
I20260926 13:40:17.748849 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:17.749158 1220 external_mini_cluster.cc:1428] Setting key 3a1bc7bb7453c9418e79b677000ee64f
I20260926 13:40:17.752563 6847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36: Bootstrap starting.
I20260926 13:40:17.755062 6847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:17.755972 6847 log.cc:824] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:17.758242 6847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36: No bootstrap required, opened a new log
Sep 26 13:40:17 dist-test-slave-1wns krb5kdc[6766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:17.762046 6847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc914116d2454e980a37a591f37b36" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40993 } }
I20260926 13:40:17.762364 6847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:17.762454 6847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bbc914116d2454e980a37a591f37b36, State: Initialized, Role: FOLLOWER
I20260926 13:40:17.763020 6847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc914116d2454e980a37a591f37b36" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40993 } }
I20260926 13:40:17.763202 6847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:17.763295 6847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:17.763437 6847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:17.764724 6847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc914116d2454e980a37a591f37b36" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40993 } }
I20260926 13:40:17.765147 6847 leader_election.cc:302] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bbc914116d2454e980a37a591f37b36; no voters:
I20260926 13:40:17.765657 6847 leader_election.cc:288] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:17.765791 6852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:17.766100 6852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [term 1 LEADER]: Becoming Leader. State: Replica: 5bbc914116d2454e980a37a591f37b36, State: Running, Role: LEADER
I20260926 13:40:17.766479 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:17.750884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.1.49.62:40993)
0926 13:40:17.751447 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:17.751463 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:17.751511 (+ 48us) server_negotiation.cc:407] Connection header received
0926 13:40:17.752209 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:17.752229 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:17.752709 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:17.753128 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:17.753988 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:17.755014 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:17.756004 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:17.756567 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:17.756745 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:17.759695 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:17.759701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:17.759718 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:17.759752 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:17.762511 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:17.763123 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:17.763130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:17.763137 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:17.763354 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:17.763726 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:17.763729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:17.763730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:17.764266 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:17.764437 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:17.764777 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 13:40:17.765195 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":183,"threads_started":1}
I20260926 13:40:17.766491 6852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc914116d2454e980a37a591f37b36" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40993 } }
I20260926 13:40:17.767316 6847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:17.768551 6853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bbc914116d2454e980a37a591f37b36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc914116d2454e980a37a591f37b36" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40993 } } }
I20260926 13:40:17.768751 6853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:17.769130 6854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bbc914116d2454e980a37a591f37b36. Latest consensus state: current_term: 1 leader_uuid: "5bbc914116d2454e980a37a591f37b36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc914116d2454e980a37a591f37b36" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40993 } } }
I20260926 13:40:17.769285 6854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:17.769598 6861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:17.773576 6861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:17.780205 6861 catalog_manager.cc:1347] Generated new cluster ID: bda569bba8f142ea8e0f88ef60c64a4a
I20260926 13:40:17.780321 6861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:17.794659 6861 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:17.795696 6861 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:17.805680 6861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bbc914116d2454e980a37a591f37b36: Generated new TSK 0
I20260926 13:40:17.806775 6861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:40:17.839215 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40993
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:17.967077 6873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:17.967478 6873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:17.967569 6873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:17.972168 6873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:17.972281 6873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:17.972386 6873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:40:17.978302 6873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:17.979583 6873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:17.980808 6873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:17.991878 6879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:17.991981 6878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:17.992095 6873 server_base.cc:1034] running on GCE node
W20260926 13:40:17.991883 6881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:17.992641 6873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:17.993391 6873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:17.994585 6873 hybrid_clock.cc:648] HybridClock initialized: now 1790430017994565 us; error 40 us; skew 500 ppm
Sep 26 13:40:17 dist-test-slave-1wns krb5kdc[6766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:17.997715 6873 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:40:17.998783 6873 webserver.cc:460] Webserver started at http://127.1.49.1:34635/ using document root <none> and password file <none>
I20260926 13:40:17.999365 6873 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:17.999437 6873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:17.999615 6873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:18.001927 6873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "7ba61ba489ed4e2db0dbb05fda22e76d"
format_stamp: "Formatted at 2026-09-26 13:40:17 on dist-test-slave-1wns"
server_key: "fa901585a04ff54b93b0109ae5aae73e"
server_key_iv: "3bf966bddc7ec2e9c091c0fd690d2411"
server_key_version: "encryptionkey@0"
I20260926 13:40:18.002722 6873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "7ba61ba489ed4e2db0dbb05fda22e76d"
format_stamp: "Formatted at 2026-09-26 13:40:17 on dist-test-slave-1wns"
server_key: "fa901585a04ff54b93b0109ae5aae73e"
server_key_iv: "3bf966bddc7ec2e9c091c0fd690d2411"
server_key_version: "encryptionkey@0"
I20260926 13:40:18.007258 6873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 13:40:18.010365 6888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:18.012176 6873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:18.012301 6873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "7ba61ba489ed4e2db0dbb05fda22e76d"
format_stamp: "Formatted at 2026-09-26 13:40:17 on dist-test-slave-1wns"
server_key: "fa901585a04ff54b93b0109ae5aae73e"
server_key_iv: "3bf966bddc7ec2e9c091c0fd690d2411"
server_key_version: "encryptionkey@0"
I20260926 13:40:18.012508 6873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:18.024775 6873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:18.028976 6873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:18.029217 6873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:18.029928 6873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:18.030862 6873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:18.030911 6873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:18.030953 6873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:18.031009 6873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:18.041875 6873 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:37517
I20260926 13:40:18.041898 7001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:37517 every 8 connection(s)
I20260926 13:40:18.042936 6873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:40:18.046962 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 6873
I20260926 13:40:18.047137 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:40:18.047430 1220 external_mini_cluster.cc:1428] Setting key d0ba3faf8a65df61b99a3ab0cf80cd14
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
I20260926 13:40:18.062814 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:18.045259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:58625 (local address 127.1.49.62:40993)
0926 13:40:18.045505 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:18.045509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:18.046399 (+ 890us) server_negotiation.cc:407] Connection header received
0926 13:40:18.048277 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:18.048282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:18.048347 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:18.048411 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:18.050051 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.050923 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.053495 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.053794 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.053937 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:18.057702 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:18.057706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:18.057708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:18.057740 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:18.059679 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:18.060517 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:18.060521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:18.060522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:18.060625 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:18.061138 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:18.061145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:18.061147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:18.061406 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:18.061626 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:18.062322 (+ 696us) server_negotiation.cc:299] Negotiation successful
0926 13:40:18.062526 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260926 13:40:18.063556 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:18.045672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:40993 (local address 127.1.49.1:58625)
0926 13:40:18.046218 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:18.046269 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:18.048040 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:18.048675 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:18.048687 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:18.049218 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:18.049843 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:18.049859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.051250 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.051255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:18.053211 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:18.053231 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.054033 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.054041 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:18.054351 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:18.055166 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:18.055193 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:18.057442 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:18.059926 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:18.059940 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:18.059958 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:18.060312 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:18.060753 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:18.060756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:18.060758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:18.060936 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:18.061523 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:18.061530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:18.061931 (+ 401us) client_negotiation.cc:769] Sending connection context
0926 13:40:18.062193 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 13:40:18.062555 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":119,"threads_started":1}
I20260926 13:40:18.065091 7002 heartbeater.cc:347] Connected to a master server at 127.1.49.62:40993
I20260926 13:40:18.065342 7002 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:18.065882 7002 heartbeater.cc:511] Master 127.1.49.62:40993 requested a full tablet report, sending...
I20260926 13:40:18.067649 6811 ts_manager.cc:194] Registered new tserver with Master: 7ba61ba489ed4e2db0dbb05fda22e76d (127.1.49.1:37517)
I20260926 13:40:18.069187 6811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:58625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:40:18.084605 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40993
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:18.206562 7007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:18.206880 7007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:18.206925 7007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:18.211305 7007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:18.211390 7007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:18.211458 7007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:40:18.217505 7007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:18.219200 7007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:18.220505 7007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:40:18.233078 7007 server_base.cc:1034] running on GCE node
W20260926 13:40:18.232985 7012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:18.233024 7013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:18.232996 7015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:18.233793 7007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:18.234638 7007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:18.235914 7007 hybrid_clock.cc:648] HybridClock initialized: now 1790430018235892 us; error 38 us; skew 500 ppm
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430018, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:18.239461 7007 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:40:18.240607 7007 webserver.cc:460] Webserver started at http://127.1.49.2:41665/ using document root <none> and password file <none>
I20260926 13:40:18.241322 7007 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:18.241393 7007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:18.241613 7007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:18.243896 7007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "5fa7efb92c0c47ce878c56f6b9c3c1b0"
format_stamp: "Formatted at 2026-09-26 13:40:18 on dist-test-slave-1wns"
server_key: "65c4b59c692e3874e6f390fc6045d6a7"
server_key_iv: "bab20a0c81c395bcd405e937f6b9be99"
server_key_version: "encryptionkey@0"
I20260926 13:40:18.244480 7007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "5fa7efb92c0c47ce878c56f6b9c3c1b0"
format_stamp: "Formatted at 2026-09-26 13:40:18 on dist-test-slave-1wns"
server_key: "65c4b59c692e3874e6f390fc6045d6a7"
server_key_iv: "bab20a0c81c395bcd405e937f6b9be99"
server_key_version: "encryptionkey@0"
I20260926 13:40:18.248805 7007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 13:40:18.251673 7022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:18.253330 7007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:18.253531 7007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "5fa7efb92c0c47ce878c56f6b9c3c1b0"
format_stamp: "Formatted at 2026-09-26 13:40:18 on dist-test-slave-1wns"
server_key: "65c4b59c692e3874e6f390fc6045d6a7"
server_key_iv: "bab20a0c81c395bcd405e937f6b9be99"
server_key_version: "encryptionkey@0"
I20260926 13:40:18.253721 7007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:18.265619 7007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:18.269599 7007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:18.269829 7007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:18.270500 7007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:18.271402 7007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:18.271451 7007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:18.271493 7007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:18.271548 7007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:18.280319 7007 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:40767
I20260926 13:40:18.280368 7135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:40767 every 8 connection(s)
I20260926 13:40:18.281337 7007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:40:18.282179 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 7007
I20260926 13:40:18.282270 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:40:18.282567 1220 external_mini_cluster.cc:1428] Setting key 4fee9fb64304125eccd9bad64a6ffc8d
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430018, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:18.300899 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:18.283424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:51961 (local address 127.1.49.62:40993)
0926 13:40:18.283563 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:18.283569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:18.284778 (+ 1209us) server_negotiation.cc:407] Connection header received
0926 13:40:18.286694 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:18.286697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:18.286751 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:18.286829 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:18.288284 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.289207 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.290408 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.290792 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.290917 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:18.296668 (+ 5751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:18.296672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:18.296674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:18.296705 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:18.298251 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:18.298952 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:18.298958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:18.298963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:18.299068 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:18.299415 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:18.299422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:18.299426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:18.299711 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:18.299857 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:18.300483 (+ 626us) server_negotiation.cc:299] Negotiation successful
0926 13:40:18.300650 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:40:18.301692 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:18.284072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:40993 (local address 127.1.49.2:51961)
0926 13:40:18.284572 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:18.284654 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:18.286393 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:18.287091 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:18.287100 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:18.287520 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:18.288118 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:18.288133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.289356 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.289359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:18.290275 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:18.290283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.290955 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.290958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:18.291151 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:18.291888 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:18.291913 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:18.296424 (+ 4511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:18.298447 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:18.298454 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:18.298470 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:18.298821 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:18.299162 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:18.299164 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:18.299166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:18.299306 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:18.299879 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:18.299885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:18.300209 (+ 324us) client_negotiation.cc:769] Sending connection context
0926 13:40:18.300376 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 13:40:18.300691 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":139,"threads_started":1}
I20260926 13:40:18.303164 7136 heartbeater.cc:347] Connected to a master server at 127.1.49.62:40993
I20260926 13:40:18.303452 7136 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:18.304075 7136 heartbeater.cc:511] Master 127.1.49.62:40993 requested a full tablet report, sending...
I20260926 13:40:18.305398 6811 ts_manager.cc:194] Registered new tserver with Master: 5fa7efb92c0c47ce878c56f6b9c3c1b0 (127.1.49.2:40767)
I20260926 13:40:18.306195 6811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:51961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:40:18.316831 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40993
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:18.443182 7141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:18.443483 7141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:18.443532 7141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:18.448096 7141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:18.448179 7141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:18.448243 7141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:40:18.454113 7141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:18.455310 7141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:18.456442 7141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:18.467223 7147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:18.467204 7149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:18.467204 7146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:18.467763 7141 server_base.cc:1034] running on GCE node
I20260926 13:40:18.468335 7141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:18.469210 7141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:18.470449 7141 hybrid_clock.cc:648] HybridClock initialized: now 1790430018470417 us; error 60 us; skew 500 ppm
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430018, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:18.473773 7141 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:40:18.474857 7141 webserver.cc:460] Webserver started at http://127.1.49.3:45619/ using document root <none> and password file <none>
I20260926 13:40:18.475474 7141 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:18.475569 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:18.475852 7141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:18.478025 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "f4c3698c26314147a52457648cafa1b6"
format_stamp: "Formatted at 2026-09-26 13:40:18 on dist-test-slave-1wns"
server_key: "5fe53a8ac2dd94f63620e5da761ded34"
server_key_iv: "67e8fa1ead77cc198bf5f55d39291f0f"
server_key_version: "encryptionkey@0"
I20260926 13:40:18.478647 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "f4c3698c26314147a52457648cafa1b6"
format_stamp: "Formatted at 2026-09-26 13:40:18 on dist-test-slave-1wns"
server_key: "5fe53a8ac2dd94f63620e5da761ded34"
server_key_iv: "67e8fa1ead77cc198bf5f55d39291f0f"
server_key_version: "encryptionkey@0"
I20260926 13:40:18.482828 7141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 13:40:18.486079 7156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:18.487774 7141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:40:18.488042 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "f4c3698c26314147a52457648cafa1b6"
format_stamp: "Formatted at 2026-09-26 13:40:18 on dist-test-slave-1wns"
server_key: "5fe53a8ac2dd94f63620e5da761ded34"
server_key_iv: "67e8fa1ead77cc198bf5f55d39291f0f"
server_key_version: "encryptionkey@0"
I20260926 13:40:18.488194 7141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:18.524585 7141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:18.528631 7141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:18.528874 7141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:18.529563 7141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:18.530516 7141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:18.530596 7141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:18.530675 7141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:18.530731 7141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:18.541148 7141 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:39497
I20260926 13:40:18.541204 7269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:39497 every 8 connection(s)
I20260926 13:40:18.542234 7141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:40:18.545046 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 7141
I20260926 13:40:18.545199 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:40:18.545523 1220 external_mini_cluster.cc:1428] Setting key 75cf10a0e8f7bedc1c0acff05c37c71e
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430018, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:18.558629 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:18.544569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:43035 (local address 127.1.49.62:40993)
0926 13:40:18.544710 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:18.544714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:18.545840 (+ 1126us) server_negotiation.cc:407] Connection header received
0926 13:40:18.547904 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:18.547907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:18.547970 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:18.548094 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:18.549416 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.549973 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.551250 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.551491 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.551591 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:18.554511 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:18.554514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:18.554516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:18.554543 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:18.556234 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:18.556798 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:18.556801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:18.556803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:18.556899 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:18.557239 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:18.557242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:18.557243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:18.557531 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:18.557630 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:18.558281 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 13:40:18.558455 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 13:40:18.560012 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:18.545049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:40993 (local address 127.1.49.3:43035)
0926 13:40:18.545635 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:18.545698 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:18.547553 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:18.548281 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:18.548289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:18.548696 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:18.549228 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:18.549243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.550111 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.550114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:18.551101 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:18.551111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.551662 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.551665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:18.551905 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:18.552623 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:18.552640 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:18.554328 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:18.556377 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:18.556385 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:18.556404 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:18.556694 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:18.556989 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:18.556991 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:18.556993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:18.557130 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:18.557698 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:18.557708 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:18.558016 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 13:40:18.558248 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 13:40:18.558655 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":147,"threads_started":1}
I20260926 13:40:18.561297 7270 heartbeater.cc:347] Connected to a master server at 127.1.49.62:40993
I20260926 13:40:18.561597 7270 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:18.562136 7270 heartbeater.cc:511] Master 127.1.49.62:40993 requested a full tablet report, sending...
I20260926 13:40:18.563330 6811 ts_manager.cc:194] Registered new tserver with Master: f4c3698c26314147a52457648cafa1b6 (127.1.49.3:39497)
I20260926 13:40:18.564081 6811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:43035
I20260926 13:40:18.572515 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/test-admin.keytab.
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:18.589272 1220 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 13:40:18.592789 7277 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:18.590911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.1.49.1:37517)
0926 13:40:18.591210 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:18.591219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:18.591250 (+ 31us) server_negotiation.cc:407] Connection header received
0926 13:40:18.591443 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:18.591449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:18.591695 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:18.591897 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:18.592461 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40538: BlockingRecv error: recv got EOF from 127.0.0.1:40538 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":65,"threads_started":1}
W20260926 13:40:18.592897 1220 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/krb5cc' of type FILE
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:18.594136 1220 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:18.595275 1220 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.1@KRBTEST.COM
I20260926 13:40:18.607125 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:18.596641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.1.49.1:37517)
0926 13:40:18.596809 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:18.596816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:18.596834 (+ 18us) server_negotiation.cc:407] Connection header received
0926 13:40:18.597036 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:18.597040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:18.597094 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:18.597184 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:18.597763 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.598539 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.599410 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.599971 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.600098 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:18.602680 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:18.602689 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:18.602705 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:18.602731 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:18.604679 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:18.605303 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:18.605310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:18.605311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:18.605399 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:18.605788 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:18.605791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:18.605793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:18.606137 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:18.606322 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:18.606778 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 13:40:18.606963 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 13:40:18.611541 7277 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:18.610419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.1.49.1:37517)
0926 13:40:18.610589 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:18.610595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:18.610610 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:40:18.610778 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:18.610781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:18.610834 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:18.610930 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:18.611414 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40556: BlockingRecv error: recv got EOF from 127.0.0.1:40556 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 13:40:18.611744 1220 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/krb5cc' of type FILE
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:18.612928 1220 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:18.613910 1220 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.1@KRBTEST.COM
I20260926 13:40:18.624527 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:18.614900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.1.49.1:37517)
0926 13:40:18.615063 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:18.615067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:18.615082 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:40:18.615221 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:18.615223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:18.615264 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:18.615341 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:18.616034 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.616712 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.617754 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.618018 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.618125 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:18.620784 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:18.620788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:18.620790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:18.620820 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:18.622561 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:18.623027 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:18.623031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:18.623033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:18.623128 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:18.623448 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:18.623452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:18.623453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:18.623663 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:18.623985 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:18.624161 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 13:40:18.624272 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 13:40:18.628010 7277 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:18.626678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.1.49.1:37517)
0926 13:40:18.626834 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:18.626839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:18.626967 (+ 128us) server_negotiation.cc:407] Connection header received
0926 13:40:18.627152 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:18.627155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:18.627207 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:18.627300 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:18.627848 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40580: BlockingRecv error: recv got EOF from 127.0.0.1:40580 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 13:40:18.628281 1220 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790429983839632-1220-0/krb5kdc/krb5cc' of type FILE
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:18.629442 1220 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:18.630414 1220 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[6766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.1@KRBTEST.COM
I20260926 13:40:18.640949 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:18.631387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40586 (local address 127.1.49.1:37517)
0926 13:40:18.631586 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:18.631590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:18.631677 (+ 87us) server_negotiation.cc:407] Connection header received
0926 13:40:18.631882 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:18.631886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:18.631955 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:18.632038 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:18.632647 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.633149 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.634067 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:18.634277 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:18.634370 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:18.637088 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:18.637091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:18.637093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:18.637121 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:18.638749 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:18.639259 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:18.639262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:18.639263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:18.639418 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:18.639922 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:18.639926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:18.639927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:18.640150 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:18.640342 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:18.640613 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 13:40:18.640738 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 13:40:18.642199 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 6873
I20260926 13:40:18.648326 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 7007
I20260926 13:40:18.655787 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 7141
I20260926 13:40:18.661594 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 6779
2026-09-26T13:40:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2390 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[7290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[7290](info): set up 2 sockets
Sep 26 13:40:18 dist-test-slave-1wns krb5kdc[7290](info): commencing operation
krb5kdc: starting...
W20260926 13:40:19.744956 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:40:19 dist-test-slave-1wns krb5kdc[7290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:40:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.62.keytab.
Entry for principal oryx/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.62.keytab.
I20260926 13:40:19.883369 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:36609
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.62.keytab
--principal=oryx/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:36609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:20.000147 7303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:20.000469 7303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:20.000514 7303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:20.005757 7303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:20.005864 7303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:20.005888 7303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:20.005908 7303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:20.005925 7303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:20.012916 7303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:36609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.62.keytab
--principal=oryx/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:36609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:20.014169 7303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:20.015411 7303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:20.024639 7309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:20.024639 7311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:20.024792 7308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:20.025051 7303 server_base.cc:1034] running on GCE node
I20260926 13:40:20.025800 7303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:20.027415 7303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:20.028681 7303 hybrid_clock.cc:648] HybridClock initialized: now 1790430020028595 us; error 105 us; skew 500 ppm
Sep 26 13:40:20 dist-test-slave-1wns krb5kdc[7290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430020, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:20.038436 7303 init.cc:377] Logged in from keytab as oryx/127.1.49.62@KRBTEST.COM (short username oryx)
I20260926 13:40:20.039633 7303 webserver.cc:460] Webserver started at http://127.1.49.62:43899/ using document root <none> and password file <none>
I20260926 13:40:20.040488 7303 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:20.040588 7303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:20.040798 7303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:20.043236 7303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "ca6d743256e747e7bd97ca28676150d4"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "f4c8281ef1c6bea03e6855b40b00e308"
server_key_iv: "556cc3ffc2042268aa9dafb45edb4eae"
server_key_version: "encryptionkey@0"
I20260926 13:40:20.044080 7303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "ca6d743256e747e7bd97ca28676150d4"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "f4c8281ef1c6bea03e6855b40b00e308"
server_key_iv: "556cc3ffc2042268aa9dafb45edb4eae"
server_key_version: "encryptionkey@0"
I20260926 13:40:20.048653 7303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 13:40:20.051901 7318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:20.053653 7303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 13:40:20.053828 7303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "ca6d743256e747e7bd97ca28676150d4"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "f4c8281ef1c6bea03e6855b40b00e308"
server_key_iv: "556cc3ffc2042268aa9dafb45edb4eae"
server_key_version: "encryptionkey@0"
I20260926 13:40:20.053987 7303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:20.077229 7303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:20.081162 7303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:20.081385 7303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:20.089104 7303 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:36609
I20260926 13:40:20.089516 7370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:36609 every 8 connection(s)
I20260926 13:40:20.090394 7303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:20.091312 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 7303
I20260926 13:40:20.091481 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:20.091784 1220 external_mini_cluster.cc:1428] Setting key dee20234dbec948a14427f9e212ac922
I20260926 13:40:20.094269 7371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:20.102346 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4: Bootstrap starting.
Sep 26 13:40:20 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
I20260926 13:40:20.105579 7371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:20.106652 7371 log.cc:824] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:20.109822 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4: No bootstrap required, opened a new log
I20260926 13:40:20.110683 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:20.093320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59810 (local address 127.1.49.62:36609)
0926 13:40:20.093816 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:20.093837 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:20.093888 (+ 51us) server_negotiation.cc:407] Connection header received
0926 13:40:20.094873 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:20.094899 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:20.095385 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:20.095865 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:20.096725 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:20.098484 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:20.099506 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:20.100360 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:20.100550 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:20.103876 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:20.103887 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:20.103927 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:20.103976 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:20.106410 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:20.107166 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:20.107174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:20.107183 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:20.107342 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:20.107965 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:20.107969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:20.107971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:20.108685 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:20.108905 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:20.109238 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 13:40:20.109597 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":141,"threads_started":1}
I20260926 13:40:20.114514 7371 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } }
I20260926 13:40:20.114814 7371 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:20.114916 7371 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6d743256e747e7bd97ca28676150d4, State: Initialized, Role: FOLLOWER
I20260926 13:40:20.115406 7371 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } }
I20260926 13:40:20.115573 7371 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:20.115665 7371 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:20.115849 7371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:20.117169 7371 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } }
I20260926 13:40:20.117559 7371 leader_election.cc:302] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca6d743256e747e7bd97ca28676150d4; no voters:
I20260926 13:40:20.117902 7371 leader_election.cc:288] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:20.118149 7376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:20.118480 7376 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [term 1 LEADER]: Becoming Leader. State: Replica: ca6d743256e747e7bd97ca28676150d4, State: Running, Role: LEADER
I20260926 13:40:20.118829 7376 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } }
I20260926 13:40:20.119319 7371 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:20.120918 7377 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } }
I20260926 13:40:20.121078 7377 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:20.121109 7378 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca6d743256e747e7bd97ca28676150d4. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } }
I20260926 13:40:20.121222 7378 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:20.121644 7387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:20.124754 7387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:20.131289 7387 catalog_manager.cc:1347] Generated new cluster ID: f166fbec81484299be1710d5dd29f21d
I20260926 13:40:20.131403 7387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:20.141316 7387 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:20.143057 7387 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:20.151520 7387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4: Generated new TSK 0
I20260926 13:40:20.152442 7387 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.1.keytab.
Entry for principal oryx/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.1.keytab.
I20260926 13:40:20.188279 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36609
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.1.keytab
--principal=oryx/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:20.318542 7397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:20.318864 7397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:20.318912 7397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:20.323740 7397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:20.323858 7397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:20.323925 7397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:40:20.329861 7397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.1.keytab
--principal=oryx/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:20.331099 7397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:20.332252 7397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:20.343361 7403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:20.343518 7402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:20.343707 7397 server_base.cc:1034] running on GCE node
W20260926 13:40:20.343518 7405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:20.344432 7397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:20.345305 7397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:20.346573 7397 hybrid_clock.cc:648] HybridClock initialized: now 1790430020346532 us; error 56 us; skew 500 ppm
Sep 26 13:40:20 dist-test-slave-1wns krb5kdc[7290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430020, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:20.350068 7397 init.cc:377] Logged in from keytab as oryx/127.1.49.1@KRBTEST.COM (short username oryx)
I20260926 13:40:20.351105 7397 webserver.cc:460] Webserver started at http://127.1.49.1:43021/ using document root <none> and password file <none>
I20260926 13:40:20.351749 7397 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:20.351902 7397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:20.352114 7397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:20.354490 7397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "04dccfca386341c08195a928a2aecd0e"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "de16939b2670457a6a551ad27d776b73"
server_key_iv: "2a960f91e0e0b75ad099bf40e863ac8f"
server_key_version: "encryptionkey@0"
I20260926 13:40:20.355088 7397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "04dccfca386341c08195a928a2aecd0e"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "de16939b2670457a6a551ad27d776b73"
server_key_iv: "2a960f91e0e0b75ad099bf40e863ac8f"
server_key_version: "encryptionkey@0"
I20260926 13:40:20.359709 7397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 13:40:20.362799 7412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:20.364421 7397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:20.364557 7397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "04dccfca386341c08195a928a2aecd0e"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "de16939b2670457a6a551ad27d776b73"
server_key_iv: "2a960f91e0e0b75ad099bf40e863ac8f"
server_key_version: "encryptionkey@0"
I20260926 13:40:20.364673 7397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:20.396179 7397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:20.400187 7397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:20.400380 7397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:20.400993 7397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:20.402112 7397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:20.402169 7397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:20.402217 7397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:20.402278 7397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:20.412546 7397 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:45087
I20260926 13:40:20.412563 7525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:45087 every 8 connection(s)
I20260926 13:40:20.413569 7397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:40:20.416659 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 7397
I20260926 13:40:20.416838 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:40:20.417222 1220 external_mini_cluster.cc:1428] Setting key f43cb9b10c5a6f50407f30f8575d4159
Sep 26 13:40:20 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430020, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.1.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.49.2@KRBTEST.COM" created.
I20260926 13:40:20.431675 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:20.415694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:41607 (local address 127.1.49.62:36609)
0926 13:40:20.415933 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:20.415937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:20.417488 (+ 1551us) server_negotiation.cc:407] Connection header received
0926 13:40:20.419128 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:20.419132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:20.419193 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:20.419304 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:20.421333 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:20.422033 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:20.423590 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:20.423977 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:20.424118 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:20.427156 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:20.427161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:20.427165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:20.427207 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:20.428972 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:20.429608 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:20.429611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:20.429612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:20.429707 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:20.430155 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:20.430158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:20.430159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:20.430374 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:20.430509 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:20.431244 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 13:40:20.431404 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:40:20.432540 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:20.416146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.1:41607)
0926 13:40:20.417256 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:20.417334 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:20.418793 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:20.419658 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:20.419666 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:20.420283 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:20.421050 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:20.421080 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:20.422195 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:20.422202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:20.423415 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:20.423427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:20.424174 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:20.424180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:20.424401 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:20.425160 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:20.425182 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:20.426916 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:20.429108 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:20.429114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:20.429129 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:20.429480 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:20.429847 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:20.429865 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:20.429866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:20.430032 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:20.430519 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:20.430526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:20.430872 (+ 346us) client_negotiation.cc:769] Sending connection context
0926 13:40:20.431064 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 13:40:20.431413 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":670,"threads_started":1}
I20260926 13:40:20.434145 7526 heartbeater.cc:347] Connected to a master server at 127.1.49.62:36609
I20260926 13:40:20.434520 7526 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:20.435132 7526 heartbeater.cc:511] Master 127.1.49.62:36609 requested a full tablet report, sending...
I20260926 13:40:20.437017 7335 ts_manager.cc:194] Registered new tserver with Master: 04dccfca386341c08195a928a2aecd0e (127.1.49.1:45087)
I20260926 13:40:20.438371 7335 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:41607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.2.keytab.
Entry for principal oryx/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.2.keytab.
I20260926 13:40:20.452090 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36609
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.2.keytab
--principal=oryx/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:20.574482 7531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:20.574779 7531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:20.574867 7531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:20.579177 7531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:20.579254 7531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:20.579316 7531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:40:20.585136 7531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.2.keytab
--principal=oryx/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:20.586333 7531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:20.587479 7531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:20.598300 7539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:20.598579 7531 server_base.cc:1034] running on GCE node
W20260926 13:40:20.598310 7537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:20.598301 7536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:20.599332 7531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:20.600239 7531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:20.601569 7531 hybrid_clock.cc:648] HybridClock initialized: now 1790430020601550 us; error 36 us; skew 500 ppm
Sep 26 13:40:20 dist-test-slave-1wns krb5kdc[7290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430020, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:20.605005 7531 init.cc:377] Logged in from keytab as oryx/127.1.49.2@KRBTEST.COM (short username oryx)
I20260926 13:40:20.605991 7531 webserver.cc:460] Webserver started at http://127.1.49.2:38575/ using document root <none> and password file <none>
I20260926 13:40:20.606642 7531 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:20.606712 7531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:20.606946 7531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:20.609381 7531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "3e0ce36f39e9402883d8942956b16c27"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "f2cb6b518a9bcb7cdd51f7d44ff76df8"
server_key_iv: "13f610e109ec18cb05bccfeaa3d3be93"
server_key_version: "encryptionkey@0"
I20260926 13:40:20.610059 7531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "3e0ce36f39e9402883d8942956b16c27"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "f2cb6b518a9bcb7cdd51f7d44ff76df8"
server_key_iv: "13f610e109ec18cb05bccfeaa3d3be93"
server_key_version: "encryptionkey@0"
I20260926 13:40:20.615415 7531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 13:40:20.618814 7546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:20.620824 7531 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260926 13:40:20.621003 7531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "3e0ce36f39e9402883d8942956b16c27"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "f2cb6b518a9bcb7cdd51f7d44ff76df8"
server_key_iv: "13f610e109ec18cb05bccfeaa3d3be93"
server_key_version: "encryptionkey@0"
I20260926 13:40:20.621186 7531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:20.639508 7531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:20.643705 7531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:20.643994 7531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:20.644712 7531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:20.645812 7531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:20.645907 7531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:20.645994 7531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:20.646044 7531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:20.656711 7531 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:34249
I20260926 13:40:20.656785 7659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:34249 every 8 connection(s)
I20260926 13:40:20.657806 7531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:40:20.659793 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 7531
I20260926 13:40:20.659960 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:40:20.660315 1220 external_mini_cluster.cc:1428] Setting key d8e1417ba0b1e156f77bddfe65dd47d2
WARNING: no policy specified for oryx/127.1.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.49.3@KRBTEST.COM" created.
Sep 26 13:40:20 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430020, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
I20260926 13:40:20.679922 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:20.660160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:54509 (local address 127.1.49.62:36609)
0926 13:40:20.660344 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:20.660349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:20.661474 (+ 1125us) server_negotiation.cc:407] Connection header received
0926 13:40:20.663656 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:20.663659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:20.663713 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:20.663928 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:20.665641 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:20.666238 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:20.667606 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:20.667965 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:20.668087 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:20.675266 (+ 7179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:20.675271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:20.675276 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:20.675387 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:20.677416 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:20.678127 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:20.678131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:20.678133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:20.678225 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:20.678650 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:20.678654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:20.678656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:20.678865 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:20.678974 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:20.679570 (+ 596us) server_negotiation.cc:299] Negotiation successful
0926 13:40:20.679716 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 13:40:20.680891 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:20.660770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.2:54509)
0926 13:40:20.661296 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:20.661366 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:20.663332 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:20.664178 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:20.664187 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:20.664847 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:20.665454 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:20.665472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:20.666402 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:20.666411 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:20.667441 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:20.667453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:20.668158 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:20.668163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:20.668358 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:20.669170 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:20.669191 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:20.675002 (+ 5811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:20.677601 (+ 2599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:20.677609 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:20.677629 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:20.678006 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:20.678360 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:20.678363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:20.678365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:20.678545 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:20.678995 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:20.679003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:20.679309 (+ 306us) client_negotiation.cc:769] Sending connection context
0926 13:40:20.679529 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 13:40:20.679952 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":147,"threads_started":1}
I20260926 13:40:20.682508 7660 heartbeater.cc:347] Connected to a master server at 127.1.49.62:36609
I20260926 13:40:20.682826 7660 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:20.683418 7660 heartbeater.cc:511] Master 127.1.49.62:36609 requested a full tablet report, sending...
I20260926 13:40:20.684763 7335 ts_manager.cc:194] Registered new tserver with Master: 3e0ce36f39e9402883d8942956b16c27 (127.1.49.2:34249)
I20260926 13:40:20.685449 7335 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:54509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.3.keytab.
Entry for principal oryx/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.3.keytab.
I20260926 13:40:20.695699 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36609
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.3.keytab
--principal=oryx/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:20.827142 7665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:20.827451 7665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:20.827525 7665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:20.831784 7665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:20.831944 7665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:20.832049 7665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:40:20.837893 7665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.3.keytab
--principal=oryx/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:20.839008 7665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:20.840070 7665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:20.852010 7673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:20.852036 7671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:20.852133 7665 server_base.cc:1034] running on GCE node
W20260926 13:40:20.852021 7670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:20.853017 7665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:20.853876 7665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:20.855132 7665 hybrid_clock.cc:648] HybridClock initialized: now 1790430020855078 us; error 61 us; skew 500 ppm
Sep 26 13:40:20 dist-test-slave-1wns krb5kdc[7290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430020, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:20.858609 7665 init.cc:377] Logged in from keytab as oryx/127.1.49.3@KRBTEST.COM (short username oryx)
I20260926 13:40:20.859850 7665 webserver.cc:460] Webserver started at http://127.1.49.3:38883/ using document root <none> and password file <none>
I20260926 13:40:20.860503 7665 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:20.860606 7665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:20.860999 7665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:20.863658 7665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "33d6d78df9f445f5852b6f10003d12da"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "c2798bfd3399944b949c2d67768ef0ee"
server_key_iv: "d37ff1371b46f6417eb977f1f391c34f"
server_key_version: "encryptionkey@0"
I20260926 13:40:20.864466 7665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "33d6d78df9f445f5852b6f10003d12da"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "c2798bfd3399944b949c2d67768ef0ee"
server_key_iv: "d37ff1371b46f6417eb977f1f391c34f"
server_key_version: "encryptionkey@0"
I20260926 13:40:20.869148 7665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 13:40:20.872123 7680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:20.873651 7665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 13:40:20.873847 7665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "33d6d78df9f445f5852b6f10003d12da"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "c2798bfd3399944b949c2d67768ef0ee"
server_key_iv: "d37ff1371b46f6417eb977f1f391c34f"
server_key_version: "encryptionkey@0"
I20260926 13:40:20.873988 7665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:20.891413 7665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:20.895148 7665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:20.895372 7665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:20.896081 7665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:20.896947 7665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:20.897020 7665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:20.897092 7665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:20.897148 7665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:20.906983 7665 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:42317
I20260926 13:40:20.907013 7793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:42317 every 8 connection(s)
I20260926 13:40:20.908026 7665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:40:20.913879 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 7665
I20260926 13:40:20.914011 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:40:20.914335 1220 external_mini_cluster.cc:1428] Setting key e853a1d719b3be61beb6074d5ca4dac4
Sep 26 13:40:20 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430020, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
I20260926 13:40:20.924700 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:20.910145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:60473 (local address 127.1.49.62:36609)
0926 13:40:20.910303 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:20.910308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:20.911351 (+ 1043us) server_negotiation.cc:407] Connection header received
0926 13:40:20.912956 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:20.912960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:20.913015 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:20.913111 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:20.914774 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:20.915731 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:20.917376 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:20.917601 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:20.917713 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:20.920746 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:20.920750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:20.920752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:20.920781 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:20.922314 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:20.922960 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:20.922963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:20.922964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:20.923041 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:20.923426 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:20.923429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:20.923431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:20.923635 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:20.923741 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:20.924382 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 13:40:20.924527 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:40:20.925999 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:20.910664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.3:60473)
0926 13:40:20.911174 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:20.911226 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:20.912701 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:20.913300 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:20.913307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:20.913882 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:20.914587 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:20.914606 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:20.915960 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:20.915962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:20.917220 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:20.917230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:20.917775 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:20.917779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:20.917966 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:20.918642 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:20.918661 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:20.920488 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:20.922473 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:20.922484 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:20.922505 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:20.922828 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:20.923142 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:20.923146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:20.923148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:20.923320 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:20.923751 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:20.923761 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:20.924121 (+ 360us) client_negotiation.cc:769] Sending connection context
0926 13:40:20.924329 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 13:40:20.924744 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":119,"threads_started":1}
I20260926 13:40:20.927582 7794 heartbeater.cc:347] Connected to a master server at 127.1.49.62:36609
I20260926 13:40:20.927951 7794 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:20.928658 7794 heartbeater.cc:511] Master 127.1.49.62:36609 requested a full tablet report, sending...
I20260926 13:40:20.929934 7335 ts_manager.cc:194] Registered new tserver with Master: 33d6d78df9f445f5852b6f10003d12da (127.1.49.3:42317)
I20260926 13:40:20.930570 7335 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:60473
I20260926 13:40:20.942118 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.49.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.49.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.49.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.61.keytab.
Entry for principal oryx/127.1.49.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.61.keytab.
I20260926 13:40:20.972070 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.61:34937
--webserver_interface=127.1.49.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.61.keytab
--principal=oryx/127.1.49.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:36609,127.1.49.61:34937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:21.104355 7799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:21.104718 7799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:21.104815 7799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:21.109450 7799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:21.109593 7799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:21.109634 7799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:21.109681 7799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:21.109725 7799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:21.115638 7799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:36609,127.1.49.61:34937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.61.keytab
--principal=oryx/127.1.49.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.61:34937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.49.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
W20260926 13:40:21.116039 7799 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.49.62:36609,127.1.49.61:34937'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 13:40:21.117081 7799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:21.118248 7799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:21.128214 7805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:21.128216 7807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:21.128415 7799 server_base.cc:1034] running on GCE node
W20260926 13:40:21.128628 7804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:21.129297 7799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:21.130676 7799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:21.131912 7799 hybrid_clock.cc:648] HybridClock initialized: now 1790430021131883 us; error 35 us; skew 500 ppm
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430021, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:21.135133 7799 init.cc:377] Logged in from keytab as oryx/127.1.49.61@KRBTEST.COM (short username oryx)
I20260926 13:40:21.136245 7799 webserver.cc:460] Webserver started at http://127.1.49.61:45257/ using document root <none> and password file <none>
I20260926 13:40:21.136933 7799 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:21.137036 7799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:21.137281 7799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:21.139626 7799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/data/instance:
uuid: "1428d554ea2b485b80bcbc9048734f52"
format_stamp: "Formatted at 2026-09-26 13:40:21 on dist-test-slave-1wns"
server_key: "a1d1026f5087a3290f3bf5cab1e32ec1"
server_key_iv: "58aebb4f5ecf3a31cd6ec9c2a6800420"
server_key_version: "encryptionkey@0"
I20260926 13:40:21.140326 7799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/wal/instance:
uuid: "1428d554ea2b485b80bcbc9048734f52"
format_stamp: "Formatted at 2026-09-26 13:40:21 on dist-test-slave-1wns"
server_key: "a1d1026f5087a3290f3bf5cab1e32ec1"
server_key_iv: "58aebb4f5ecf3a31cd6ec9c2a6800420"
server_key_version: "encryptionkey@0"
I20260926 13:40:21.144498 7799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:40:21.147382 7814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:21.149210 7799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:21.149391 7799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/wal
uuid: "1428d554ea2b485b80bcbc9048734f52"
format_stamp: "Formatted at 2026-09-26 13:40:21 on dist-test-slave-1wns"
server_key: "a1d1026f5087a3290f3bf5cab1e32ec1"
server_key_iv: "58aebb4f5ecf3a31cd6ec9c2a6800420"
server_key_version: "encryptionkey@0"
I20260926 13:40:21.149539 7799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:21.170383 7799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:21.174898 7799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:21.175253 7799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:21.183358 7799 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.61:34937
I20260926 13:40:21.183388 7866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.61:34937 every 8 connection(s)
I20260926 13:40:21.184477 7799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/data/info.pb
I20260926 13:40:21.187523 7867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:21.189936 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 7799
I20260926 13:40:21.191823 7867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430021, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.61@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.49.61@KRBTEST.COM
I20260926 13:40:21.206113 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.193795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.1.49.62:36609)
0926 13:40:21.193924 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.193929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.194366 (+ 437us) server_negotiation.cc:407] Connection header received
0926 13:40:21.194616 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.194619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.194666 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.194742 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.195588 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.196194 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.197657 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.198007 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.198130 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.201499 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.201511 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.201515 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.201549 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.203535 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.204243 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.204248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.204253 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.204344 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.204772 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.204775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.204776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.205089 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.205166 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.205796 (+ 630us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.205941 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:40:21.207110 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.193726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.0.0.1:59822)
0926 13:40:21.194217 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.194278 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.194477 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.194834 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.194840 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.195212 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.195377 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.195387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.196328 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.196336 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.197528 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.197540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.198166 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.198168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.198329 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.199302 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.199329 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.201241 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.203674 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.203683 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.203692 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.204136 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.204449 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.204453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.204455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.204665 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.205213 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.205223 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.205526 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.205736 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.206103 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":175,"threads_started":1}
I20260926 13:40:21.207114 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.191634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.1.49.61:34937)
0926 13:40:21.192177 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.192194 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.192236 (+ 42us) server_negotiation.cc:407] Connection header received
0926 13:40:21.193088 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.193107 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.193584 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.193993 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.195004 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.196399 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.197626 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.198233 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.198339 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.202047 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.202058 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.202075 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.202107 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.204178 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.204787 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.204792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.204794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.204868 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.205240 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.205242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.205243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.205510 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.205646 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.206035 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.206329 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"spinlock_wait_cycles":153344,"thread_start_us":190,"threads_started":1}
I20260926 13:40:21.208760 7867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430021, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.61@KRBTEST.COM for oryx/127.1.49.61@KRBTEST.COM
I20260926 13:40:21.218540 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.209306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:34937 (local address 127.0.0.1:49134)
0926 13:40:21.209518 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.209536 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.209633 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.209928 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.209931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.210101 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.210197 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.210203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.211512 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.211514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.212160 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.212168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.212736 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.212739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.212814 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.213437 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.213451 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.214929 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.216969 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.216971 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.216973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.217174 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.217531 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.217534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.217535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.217631 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.218069 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.218072 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.218158 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.218259 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.218377 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 13:40:21.218565 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.209356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49134 (local address 127.1.49.61:34937)
0926 13:40:21.209531 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.209535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.209586 (+ 51us) server_negotiation.cc:407] Connection header received
0926 13:40:21.209768 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.209770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.209807 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.209886 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.210391 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.211352 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.212269 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.212618 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.212711 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.215074 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.215077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.215080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.215102 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.216841 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.217285 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.217289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.217290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.217411 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.217731 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.217734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.217735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.217935 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.218022 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.218337 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.218442 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:40:21.222461 7867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: Bootstrap starting.
I20260926 13:40:21.224916 7867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:21.225716 7867 log.cc:824] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:21.227937 7867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: No bootstrap required, opened a new log
I20260926 13:40:21.230880 7867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } }
I20260926 13:40:21.231087 7867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:21.231125 7867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1428d554ea2b485b80bcbc9048734f52, State: Initialized, Role: FOLLOWER
I20260926 13:40:21.231523 7867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } }
I20260926 13:40:21.232062 7873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } } }
I20260926 13:40:21.232223 7873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:40:21.232808 7867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 13:40:21.238061 7884 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:40:21.238183 7884 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:40:21.241115 7799 master_runner.cc:428] Detected that this master 1428d554ea2b485b80bcbc9048734f52 is joining an existing cluster
I20260926 13:40:21.241207 7799 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.49.61:34937
I20260926 13:40:21.252317 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.243586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.1.49.62:36609)
0926 13:40:21.243740 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.243746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.243955 (+ 209us) server_negotiation.cc:407] Connection header received
0926 13:40:21.244079 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.244082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.244146 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.244231 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.244909 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.245729 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.246892 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.247232 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.247348 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.248869 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.248873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.248876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.248903 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.250542 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.251041 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.251044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.251047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.251140 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.251451 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.251454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.251456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.251651 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.251755 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.252048 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.252162 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 13:40:21.252509 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.244201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.1.49.61:34937)
0926 13:40:21.244298 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.244302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.244448 (+ 146us) server_negotiation.cc:407] Connection header received
0926 13:40:21.244545 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.244548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.244591 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.244675 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.245118 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.245594 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.247222 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.247630 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.247734 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.249027 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.249030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.249033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.249056 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.250873 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.251296 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.251299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.251301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.251374 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.251629 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.251632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.251634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.251886 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.251953 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.252238 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.252353 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":40,"spinlock_wait_cycles":2944}
I20260926 13:40:21.252344 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.243495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.0.0.1:59836)
0926 13:40:21.243855 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.243872 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.243984 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.244364 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.244369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.244607 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.244773 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.244780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.245854 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.245857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.246751 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.246763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.247367 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.247371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.247559 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.248175 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.248190 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.248702 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.250684 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.250688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.250690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.250955 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.251231 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.251237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.251241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.251347 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.251754 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.251757 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.251872 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.251985 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.252139 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":47,"spinlock_wait_cycles":1280,"thread_start_us":118,"threads_started":1}
I20260926 13:40:21.252929 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.244014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:34937 (local address 127.0.0.1:49144)
0926 13:40:21.244357 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.244369 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.244460 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.244703 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.244706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.244908 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.245017 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.245022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.245703 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.245706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.247116 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.247123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.247876 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.247879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.247993 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.248429 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.248444 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.248900 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.250981 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.250987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.250992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.251219 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.251466 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.251468 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.251470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.251547 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.252008 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.252011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.252084 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.252640 (+ 556us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.252772 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":83,"threads_started":1}
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430020, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.62@KRBTEST.COM for oryx/127.1.49.61@KRBTEST.COM
I20260926 13:40:21.269243 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.258142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.1.49.61:34937)
0926 13:40:21.258280 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.258285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.258684 (+ 399us) server_negotiation.cc:407] Connection header received
0926 13:40:21.258964 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.258968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.259056 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.259154 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.259856 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.260620 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.261963 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.262360 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.262496 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.265138 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.265142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.265145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.265174 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.266897 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.267461 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.267464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.267466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.267558 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.268039 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.268043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.268045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.268274 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.268391 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.268871 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.269026 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:40:21.269281 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.258142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:34937 (local address 127.0.0.1:49156)
0926 13:40:21.258542 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.258591 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.258797 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.259177 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.259181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.259560 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.259690 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.259696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.260759 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.260763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.261821 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.261840 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.262622 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.262625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.262708 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.263312 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.263324 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.264978 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.267090 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.267100 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.267106 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.267360 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.267774 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.267780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.267785 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.267936 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.268455 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.268464 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.268608 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.268784 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.269090 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":128,"threads_started":1}
I20260926 13:40:21.271595 7335 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.49.61:34937: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1428d554ea2b485b80bcbc9048734f52"
member_type: NON_VOTER
last_known_addr {
host: "127.1.49.61"
port: 34937
}
attrs {
promote: true
}
}
dest_uuid: "ca6d743256e747e7bd97ca28676150d4"
cas_config_opid_index: -1
I20260926 13:40:21.272405 7335 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: NON_VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.1.49.60@KRBTEST.COM; defaulting to no policy
I20260926 13:40:21.274977 7896 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1428d554ea2b485b80bcbc9048734f52 (127.1.49.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: NON_VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: true } } }
I20260926 13:40:21.275723 7841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:21.277139 7896 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.49.61:34937
I20260926 13:40:21.277287 7841 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 FOLLOWER]: Refusing update from remote peer ca6d743256e747e7bd97ca28676150d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 13:40:21.277412 7896 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: NON_VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: true } } }
I20260926 13:40:21.277537 7896 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.49.60@KRBTEST.COM" created.
I20260926 13:40:21.278234 7799 master.cc:537] Master@127.1.49.61:34937 shutting down...
I20260926 13:40:21.278097 7895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: NON_VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.002s
I20260926 13:40:21.283490 7799 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:21.283870 7799 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:21.284065 7799 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: stopping tablet replica
W20260926 13:40:21.284672 7321 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 13:40:21.285576 7321 consensus_peers.cc:487] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 -> Peer 1428d554ea2b485b80bcbc9048734f52 (127.1.49.61:34937): Couldn't send request to peer 1428d554ea2b485b80bcbc9048734f52. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 13:40:21.289762 7799 master.cc:552] Master@127.1.49.61:34937 shutdown complete.
I20260926 13:40:21.289829 7799 master_runner.cc:305] Clearing existing system tablet
I20260926 13:40:21.290053 7799 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:21.290124 7799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:21.291230 7799 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:40:21.292279 7901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:21.292694 7799 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:21.292897 7799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:21.292965 7799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/wal
uuid: "1428d554ea2b485b80bcbc9048734f52"
format_stamp: "Formatted at 2026-09-26 13:40:21 on dist-test-slave-1wns"
server_key: "a1d1026f5087a3290f3bf5cab1e32ec1"
server_key_iv: "58aebb4f5ecf3a31cd6ec9c2a6800420"
server_key_version: "encryptionkey@0"
I20260926 13:40:21.296018 7799 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:21.300397 7799 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 13:40:21.300559 7799 log.cc:1196] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 13:40:21.300907 7799 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: Deleting consensus metadata
I20260926 13:40:21.301126 7799 master_runner.cc:315] Copying system tablet from 127.1.49.62:36609
I20260926 13:40:21.302434 7799 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: tablet copy: Beginning tablet copy session from remote peer at address 127.1.49.62:36609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.49.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.60.keytab.
Entry for principal oryx/127.1.49.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.60.keytab.
I20260926 13:40:21.310948 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.60:34631
--webserver_interface=127.1.49.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.60.keytab
--principal=oryx/127.1.49.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:36609,127.1.49.61:34937,127.1.49.60:34631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:40:21.322085 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.312385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.1.49.62:36609)
0926 13:40:21.312543 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.312548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.314052 (+ 1504us) server_negotiation.cc:407] Connection header received
0926 13:40:21.314241 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.314244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.314302 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.314400 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.314883 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.316060 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.316976 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.317209 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.317328 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.318552 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.318555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.318558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.318584 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.320284 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.320775 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.320779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.320780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.320897 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.321190 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.321194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.321196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.321399 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.321509 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.321787 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.321914 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 13:40:21.322098 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.312322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.0.0.1:59852)
0926 13:40:21.313927 (+ 1605us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.313942 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.314126 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.314417 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.314420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.314646 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.314757 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.314763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.316205 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.316210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.316833 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.316843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.317345 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.317349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.317448 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.318033 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.318046 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.318398 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.320414 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.320417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.320420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.320671 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.321012 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.321014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.321015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.321092 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.321497 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.321500 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.321613 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.321730 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.321903 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1518,"thread_start_us":120,"threads_started":1}
I20260926 13:40:21.322986 7355 tablet_copy_service.cc:140] P ca6d743256e747e7bd97ca28676150d4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1428d554ea2b485b80bcbc9048734f52 ({username='oryx', principal='oryx/127.1.49.61@KRBTEST.COM'} at 127.0.0.1:59852)
I20260926 13:40:21.323179 7355 tablet_copy_service.cc:161] P ca6d743256e747e7bd97ca28676150d4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1428d554ea2b485b80bcbc9048734f52 at {username='oryx', principal='oryx/127.1.49.61@KRBTEST.COM'} at 127.0.0.1:59852: session id = 1428d554ea2b485b80bcbc9048734f52-00000000000000000000000000000000
I20260926 13:40:21.324798 7355 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 13:40:21.326606 7799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 13:40:21.329149 7799 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: tablet copy: Starting download of 0 data blocks...
I20260926 13:40:21.329380 7799 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: tablet copy: Starting download of 1 WAL segments...
I20260926 13:40:21.331434 7799 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 13:40:21.333678 7355 tablet_copy_service.cc:342] P ca6d743256e747e7bd97ca28676150d4: Request end of tablet copy session 1428d554ea2b485b80bcbc9048734f52-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.49.61@KRBTEST.COM'} at 127.0.0.1:59852
I20260926 13:40:21.333842 7355 tablet_copy_service.cc:433] P ca6d743256e747e7bd97ca28676150d4: ending tablet copy session 1428d554ea2b485b80bcbc9048734f52-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1428d554ea2b485b80bcbc9048734f52
I20260926 13:40:21.342968 7799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:21.344880 7912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:21.345069 7910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:21.345319 7799 server_base.cc:1034] running on GCE node
W20260926 13:40:21.348429 7909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:21.348688 7799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:21.349028 7799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:21.350164 7799 hybrid_clock.cc:648] HybridClock initialized: now 1790430021350152 us; error 32 us; skew 500 ppm
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430021, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:21.352017 7799 init.cc:377] Logged in from keytab as oryx/127.1.49.61@KRBTEST.COM (short username oryx)
I20260926 13:40:21.352521 7799 webserver.cc:460] Webserver started at http://127.1.49.61:44275/ using document root <none> and password file <none>
I20260926 13:40:21.352718 7799 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:21.352759 7799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:21.354110 7799 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:40:21.355109 7919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:21.355368 7799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:40:21.355448 7799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/wal
uuid: "1428d554ea2b485b80bcbc9048734f52"
format_stamp: "Formatted at 2026-09-26 13:40:21 on dist-test-slave-1wns"
server_key: "a1d1026f5087a3290f3bf5cab1e32ec1"
server_key_iv: "58aebb4f5ecf3a31cd6ec9c2a6800420"
server_key_version: "encryptionkey@0"
I20260926 13:40:21.355530 7799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:21.374663 7799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:21.376976 7799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:21.382275 7799 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.61:34937
I20260926 13:40:21.382308 7971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.61:34937 every 8 connection(s)
I20260926 13:40:21.382869 7799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-1/data/info.pb
I20260926 13:40:21.385540 7972 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:40:21.386653 7972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: Bootstrap starting.
I20260926 13:40:21.397432 7972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:40:21.397761 7972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: Bootstrap complete.
I20260926 13:40:21.398362 7972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: NON_VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: true } }
I20260926 13:40:21.398520 7972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1428d554ea2b485b80bcbc9048734f52, State: Initialized, Role: LEARNER
I20260926 13:40:21.398646 7972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: NON_VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: true } }
I20260926 13:40:21.399060 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: NON_VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: true } } }
I20260926 13:40:21.399240 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: This master's current role is: LEARNER
I20260926 13:40:21.399256 7972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:21.403093 7986 catalog_manager.cc:1259] Loaded cluster ID: f166fbec81484299be1710d5dd29f21d
I20260926 13:40:21.403173 7986 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: loading cluster ID for follower catalog manager: success
I20260926 13:40:21.405421 7986 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: acquiring CA information for follower catalog manager: success
I20260926 13:40:21.406809 7986 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:40:21.441588 7526 heartbeater.cc:503] Master 127.1.49.62:36609 was elected leader, sending a full tablet report...
W20260926 13:40:21.457863 7907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:21.458230 7907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:21.458278 7907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:21.462936 7907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:21.463022 7907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:21.463043 7907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:21.463060 7907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:21.463076 7907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:21.469187 7907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:36609,127.1.49.61:34937,127.1.49.60:34631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.60.keytab
--principal=oryx/127.1.49.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.60:34631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.49.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:21.470460 7907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:21.471489 7907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:21.480701 7992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:21.480757 7991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:21.480782 7907 server_base.cc:1034] running on GCE node
W20260926 13:40:21.480720 7994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:21.481606 7907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:21.482774 7907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:21.483987 7907 hybrid_clock.cc:648] HybridClock initialized: now 1790430021483949 us; error 51 us; skew 500 ppm
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430021, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:21.487063 7907 init.cc:377] Logged in from keytab as oryx/127.1.49.60@KRBTEST.COM (short username oryx)
I20260926 13:40:21.488142 7907 webserver.cc:460] Webserver started at http://127.1.49.60:44673/ using document root <none> and password file <none>
I20260926 13:40:21.488727 7907 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:21.488795 7907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:21.489125 7907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:21.491539 7907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/data/instance:
uuid: "753d614d1cb643beaa988ad729abba2e"
format_stamp: "Formatted at 2026-09-26 13:40:21 on dist-test-slave-1wns"
server_key: "a1c026074f32b0360cb7d85d3612c384"
server_key_iv: "55b37ae01bf6914ad7fed4721afa6722"
server_key_version: "encryptionkey@0"
I20260926 13:40:21.492179 7907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/wal/instance:
uuid: "753d614d1cb643beaa988ad729abba2e"
format_stamp: "Formatted at 2026-09-26 13:40:21 on dist-test-slave-1wns"
server_key: "a1c026074f32b0360cb7d85d3612c384"
server_key_iv: "55b37ae01bf6914ad7fed4721afa6722"
server_key_version: "encryptionkey@0"
I20260926 13:40:21.496166 7907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:40:21.498807 8001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:21.500386 7907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:21.500543 7907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/wal
uuid: "753d614d1cb643beaa988ad729abba2e"
format_stamp: "Formatted at 2026-09-26 13:40:21 on dist-test-slave-1wns"
server_key: "a1c026074f32b0360cb7d85d3612c384"
server_key_iv: "55b37ae01bf6914ad7fed4721afa6722"
server_key_version: "encryptionkey@0"
I20260926 13:40:21.500634 7907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:21.517797 7907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:21.521462 7907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:21.521641 7907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:21.529896 7907 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.60:34631
I20260926 13:40:21.530190 8053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.60:34631 every 8 connection(s)
I20260926 13:40:21.531021 7907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/data/info.pb
I20260926 13:40:21.531693 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 7907
I20260926 13:40:21.534543 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:21.539389 8054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.49.60@KRBTEST.COM
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430021, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.60@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
I20260926 13:40:21.548919 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.533223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.1.49.60:34631)
0926 13:40:21.533628 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.533646 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.533690 (+ 44us) server_negotiation.cc:407] Connection header received
0926 13:40:21.534562 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.534589 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.535026 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.535390 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.536176 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.537699 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.538649 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.539333 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.539529 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.543013 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.543018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.543027 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.543055 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.545186 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.545735 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.545748 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.545754 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.545887 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.546221 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.546227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.546235 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.546718 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.547090 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.547438 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.547795 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":101,"threads_started":1}
I20260926 13:40:21.552891 8057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.541496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.0.0.1:59866)
0926 13:40:21.542041 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.542103 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.542256 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.542608 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.542615 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.543135 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.543274 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.543312 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.544164 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.544170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.544897 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.544906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.545386 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.545389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.545517 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.546046 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.546058 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.547898 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.550936 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.550943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.550948 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.551189 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.551581 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.551584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.551586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.551693 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.552232 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.552238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.552360 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.552495 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.552714 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":172,"threads_started":1}
I20260926 13:40:21.552896 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.541769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.1.49.62:36609)
0926 13:40:21.541912 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.541916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.542199 (+ 283us) server_negotiation.cc:407] Connection header received
0926 13:40:21.542389 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.542392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.542436 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.542507 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.543447 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.544044 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.545037 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.545260 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.545382 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.548068 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.548082 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.548084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.548110 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.550802 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.551325 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.551328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.551330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.551445 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.551893 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.551896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.551898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.552110 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.552203 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.552576 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.552760 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 13:40:21.554188 8054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430021, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.60@KRBTEST.COM for oryx/127.1.49.61@KRBTEST.COM
I20260926 13:40:21.563380 8057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.554707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:34937 (local address 127.0.0.1:49158)
0926 13:40:21.554864 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.554880 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.554962 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.555353 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.555355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.555537 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.555635 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.555640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.556722 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.556725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.557293 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.557301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.557982 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.557985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.558068 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.558630 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.558649 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.560012 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.561738 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.561740 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.561742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.561925 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.562203 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.562206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.562208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.562380 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.562861 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.562864 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.562953 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.563050 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.563173 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 13:40:21.563390 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.554856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.1.49.61:34937)
0926 13:40:21.555154 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.555158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.555170 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:40:21.555208 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.555211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.555250 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.555328 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.555751 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.556619 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.557399 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.557827 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.557965 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.560139 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.560143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.560145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.560169 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.561637 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.562014 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.562017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.562018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.562094 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.562560 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.562563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.562564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.562751 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.562841 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.563139 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.563259 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":140,"threads_started":1}
I20260926 13:40:21.564333 8054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430021, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.60@KRBTEST.COM for oryx/127.1.49.60@KRBTEST.COM
I20260926 13:40:21.573339 8057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.564818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:34631 (local address 127.0.0.1:52956)
0926 13:40:21.564976 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.564994 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.565078 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.565377 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.565379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.565512 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.565584 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.565589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.566602 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.566605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.567204 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.567214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.567669 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.567672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.567733 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.568265 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.568280 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.569649 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.571674 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.571676 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.571678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.571995 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.572382 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.572384 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.572385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.572464 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.572856 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.572859 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.572977 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.573070 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.573201 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 13:40:21.573431 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.564931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.1.49.60:34631)
0926 13:40:21.565057 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.565060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.565071 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:40:21.565171 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.565173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.565206 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.565268 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.565681 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.566493 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.567307 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.567552 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.567627 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.569811 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.569814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.569817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.569840 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.571553 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.572129 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.572133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.572136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.572238 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.572580 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.572583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.572584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.572761 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.572841 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.573123 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.573278 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":67}
I20260926 13:40:21.576887 8054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: Bootstrap starting.
I20260926 13:40:21.579453 8054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:21.580276 8054 log.cc:824] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:21.582361 8054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: No bootstrap required, opened a new log
I20260926 13:40:21.585769 8054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } }
I20260926 13:40:21.586011 8054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:21.586050 8054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 753d614d1cb643beaa988ad729abba2e, State: Initialized, Role: FOLLOWER
I20260926 13:40:21.586527 8054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } }
I20260926 13:40:21.586977 8060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } } }
I20260926 13:40:21.587127 8060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:40:21.587867 8054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [sys.catalog]: configured and running, proceeding with master startup.
W20260926 13:40:21.593021 8071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:40:21.593168 8071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:40:21.597190 7907 master_runner.cc:428] Detected that this master 753d614d1cb643beaa988ad729abba2e is joining an existing cluster
I20260926 13:40:21.597270 7907 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.49.60:34631
I20260926 13:40:21.604554 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 7397
I20260926 13:40:21.609794 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.599859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.1.49.60:34631)
0926 13:40:21.599944 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.599948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.600006 (+ 58us) server_negotiation.cc:407] Connection header received
0926 13:40:21.600168 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.600171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.600231 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.600281 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.601200 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.601962 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.603642 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.604031 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.604111 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.606083 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.606086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.606089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.606109 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.607924 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.608513 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.608517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.608518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.608594 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.608870 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.608873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.608875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.609125 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.609192 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.609459 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.609605 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":35}
I20260926 13:40:21.609815 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.599410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:34631 (local address 127.0.0.1:52966)
0926 13:40:21.599924 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.599939 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.600067 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.600460 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.600465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.600831 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.601094 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.601099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.602099 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.602102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.603534 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.603542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.604235 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.604238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.604316 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.605468 (+ 1152us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.605482 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.605931 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.608049 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.608052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.608054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.608402 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.608695 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.608698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.608703 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.608782 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.609237 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.609239 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.609303 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.609405 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.609557 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":103,"thread_start_us":134,"threads_started":1}
I20260926 13:40:21.613318 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.599454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.1.49.62:36609)
0926 13:40:21.599602 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.599607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.600131 (+ 524us) server_negotiation.cc:407] Connection header received
0926 13:40:21.600281 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.600284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.600326 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.600377 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.601242 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.602166 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.605252 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.605645 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.605755 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.608773 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.608778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.608781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.608812 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.611510 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.612019 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.612023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.612025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.612116 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.612421 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.612424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.612426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.612637 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.612747 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.613012 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.613142 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:40:21.613340 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.599612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.0.0.1:59872)
0926 13:40:21.600061 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.600072 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.600180 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.600524 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.600527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.600831 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.601079 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.601088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.603101 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.603104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.604252 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.604263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.607534 (+ 3271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.607538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.607639 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.608121 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.608139 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.608622 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.611635 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.611641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.611644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.611901 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.612221 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.612226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.612228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.612312 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.612745 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.612748 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.612843 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.612950 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.613142 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":54,"spinlock_wait_cycles":2176,"thread_start_us":101,"threads_started":1}
I20260926 13:40:21.615357 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:45087
--local_ip_for_outbound_sockets=127.1.49.1
--tserver_master_addrs=127.1.49.62:36609
--webserver_port=43021
--webserver_interface=127.1.49.1
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.1.keytab
--principal=oryx/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.49.62:36609,127.1.49.61:34937
--tserver_master_addrs=127.1.49.62:36609,127.1.49.61:34937,127.1.49.60:34631 with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430020, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.62@KRBTEST.COM for oryx/127.1.49.60@KRBTEST.COM
I20260926 13:40:21.640965 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.619057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:34631 (local address 127.0.0.1:52972)
0926 13:40:21.619256 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.619267 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.619334 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.619727 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.619730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.619978 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.620069 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.620075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.620908 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.620913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.621489 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.621497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.621972 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.621975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.622064 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.622578 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.622590 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.623960 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.637920 (+ 13960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.637928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.637932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.638263 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.638615 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.638618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.638620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.638720 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.639564 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.639569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.639698 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.640552 (+ 854us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.640750 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260926 13:40:21.641968 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.619125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.1.49.60:34631)
0926 13:40:21.619506 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.619509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.619520 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:40:21.619580 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.619582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.619624 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.619711 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.620188 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.620802 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.621629 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.621852 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.621962 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.624098 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.624104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.624111 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.624136 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.637747 (+ 13611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.638407 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.638411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.638426 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.638509 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.638830 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.638835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.638836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.639417 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.639731 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.641635 (+ 1904us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.641806 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296}
I20260926 13:40:21.642388 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.600078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.1.49.61:34937)
0926 13:40:21.600235 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.600239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.601414 (+ 1175us) server_negotiation.cc:407] Connection header received
0926 13:40:21.601585 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.601588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.601635 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.601681 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.602244 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.603020 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.606074 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.606479 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.606583 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.638622 (+ 32039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.638628 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.638630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.638667 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.640372 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.640887 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.640891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.640893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.640996 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.641396 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.641399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.641401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.641638 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.641711 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.642104 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.642235 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 13:40:21.642905 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.600327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:34937 (local address 127.0.0.1:49168)
0926 13:40:21.601334 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.601347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.601483 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.601797 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.601799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.601999 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.602122 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.602129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.603184 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.603188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.605956 (+ 2768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.605962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.606739 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.606743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.606844 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.637968 (+ 31124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.637985 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.638414 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.640502 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.640508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.640511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.640749 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.641202 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.641204 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.641206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.641287 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.641786 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.641789 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.641893 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.642562 (+ 669us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.642730 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":1783,"thread_start_us":83,"threads_started":1}
I20260926 13:40:21.643565 7335 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.49.60:34631: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "753d614d1cb643beaa988ad729abba2e"
member_type: NON_VOTER
last_known_addr {
host: "127.1.49.60"
port: 34631
}
attrs {
promote: true
}
}
dest_uuid: "ca6d743256e747e7bd97ca28676150d4"
cas_config_opid_index: 5
I20260926 13:40:21.644292 7335 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: NON_VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: true } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } }
I20260926 13:40:21.645567 7896 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 753d614d1cb643beaa988ad729abba2e (127.1.49.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: NON_VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: true } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } } }
I20260926 13:40:21.646440 8028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:21.647388 7896 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.49.60:34631
I20260926 13:40:21.647604 7896 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: NON_VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: true } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } } }
I20260926 13:40:21.647704 7896 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:21.648224 7907 master.cc:537] Master@127.1.49.60:34631 shutting down...
I20260926 13:40:21.648705 8028 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 1 FOLLOWER]: Refusing update from remote peer ca6d743256e747e7bd97ca28676150d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 13:40:21.650100 7895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 13:40:21.653330 7319 consensus_peers.cc:487] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 -> Peer 753d614d1cb643beaa988ad729abba2e (127.1.49.60:34631): Couldn't send request to peer 753d614d1cb643beaa988ad729abba2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 13:40:21.654142 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.645563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:34937 (local address 127.0.0.1:49178)
0926 13:40:21.645668 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.645685 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.645765 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.649641 (+ 3876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.649644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.649665 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:21.649761 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.649768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.650899 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.650903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.652330 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.652342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.653676 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.653680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.653779 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.653830 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.653914 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.654006 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 13:40:21.654613 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.645742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.1.49.61:34937)
0926 13:40:21.645970 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.645974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.645986 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:40:21.649343 (+ 3357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.649347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.649419 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.649509 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:21.649880 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.650674 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.652480 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.653196 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.653288 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.653570 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.654401 (+ 831us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.654488 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260926 13:40:21.655508 7944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 LEARNER]: Refusing update from remote peer ca6d743256e747e7bd97ca28676150d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 13:40:21.656096 7895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: NON_VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 13:40:21.662379 7907 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:21.662724 7907 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:21.662828 7907 tablet_replica.cc:330] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: stopping tablet replica
I20260926 13:40:21.663489 7944 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 753d614d1cb643beaa988ad729abba2e (127.1.49.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: NON_VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: true } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } } }
I20260926 13:40:21.665989 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca6d743256e747e7bd97ca28676150d4. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: NON_VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: true } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } } }
I20260926 13:40:21.666126 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: This master's current role is: LEARNER
I20260926 13:40:21.666225 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: NON_VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: true } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } } }
I20260926 13:40:21.666294 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: This master's current role is: LEARNER
I20260926 13:40:21.666877 7896 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4: attempting to promote NON_VOTER 1428d554ea2b485b80bcbc9048734f52 to VOTER
I20260926 13:40:21.667416 7896 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } }
I20260926 13:40:21.669001 7944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 LEARNER]: Refusing update from remote peer ca6d743256e747e7bd97ca28676150d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260926 13:40:21.669425 7319 consensus_peers.cc:487] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 -> Peer 753d614d1cb643beaa988ad729abba2e (127.1.49.60:34631): Couldn't send request to peer 753d614d1cb643beaa988ad729abba2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 13:40:21.669615 7895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 13:40:21.671521 7895 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1428d554ea2b485b80bcbc9048734f52 (127.1.49.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } } }
I20260926 13:40:21.672242 7944 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1428d554ea2b485b80bcbc9048734f52 (127.1.49.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } } }
I20260926 13:40:21.673202 7895 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } } }
I20260926 13:40:21.673313 7895 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:21.673429 7895 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } } }
I20260926 13:40:21.673496 7895 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:21.674005 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } } }
I20260926 13:40:21.674149 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:40:21.686472 7907 master.cc:552] Master@127.1.49.60:34631 shutdown complete.
I20260926 13:40:21.686549 7907 master_runner.cc:305] Clearing existing system tablet
I20260926 13:40:21.686818 7907 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:21.686882 7907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:21.688241 7660 heartbeater.cc:503] Master 127.1.49.62:36609 was elected leader, sending a full tablet report...
I20260926 13:40:21.688416 7907 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:40:21.689302 8085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:21.689493 7907 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:21.689600 7907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:21.689633 7907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/wal
uuid: "753d614d1cb643beaa988ad729abba2e"
format_stamp: "Formatted at 2026-09-26 13:40:21 on dist-test-slave-1wns"
server_key: "a1c026074f32b0360cb7d85d3612c384"
server_key_iv: "55b37ae01bf6914ad7fed4721afa6722"
server_key_version: "encryptionkey@0"
I20260926 13:40:21.691963 7907 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:21.694778 7907 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 13:40:21.694919 7907 log.cc:1196] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 13:40:21.695240 7907 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: Deleting consensus metadata
I20260926 13:40:21.695437 7907 master_runner.cc:315] Copying system tablet from 127.1.49.62:36609
I20260926 13:40:21.696770 7907 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: tablet copy: Beginning tablet copy session from remote peer at address 127.1.49.62:36609
I20260926 13:40:21.707177 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.697454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.1.49.62:36609)
0926 13:40:21.697591 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.697595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.697879 (+ 284us) server_negotiation.cc:407] Connection header received
0926 13:40:21.698003 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.698006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.698057 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.698157 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.698851 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.699676 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.701058 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.701454 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.701572 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.703045 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.703049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.703053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.703081 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.705073 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.705645 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.705652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.705655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.705760 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.706099 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.706103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.706104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.706329 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.706426 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.706844 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.706997 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 13:40:21.707227 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.697347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.0.0.1:59886)
0926 13:40:21.697739 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.697757 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.697888 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.698184 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.698187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.698456 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.698615 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.698627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.699852 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.699856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.700901 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.700914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.701599 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.701604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.701714 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.702348 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.702365 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.702869 (+ 504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.705215 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.705220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.705222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.705525 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.705880 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.705884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.705886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.705996 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.706502 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.706506 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.706648 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.706781 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.707008 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":126,"threads_started":1}
I20260926 13:40:21.708026 7355 tablet_copy_service.cc:140] P ca6d743256e747e7bd97ca28676150d4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 753d614d1cb643beaa988ad729abba2e ({username='oryx', principal='oryx/127.1.49.60@KRBTEST.COM'} at 127.0.0.1:59886)
I20260926 13:40:21.708177 7355 tablet_copy_service.cc:161] P ca6d743256e747e7bd97ca28676150d4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 753d614d1cb643beaa988ad729abba2e at {username='oryx', principal='oryx/127.1.49.60@KRBTEST.COM'} at 127.0.0.1:59886: session id = 753d614d1cb643beaa988ad729abba2e-00000000000000000000000000000000
I20260926 13:40:21.709398 7355 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 13:40:21.710863 7907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 13:40:21.713451 7907 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: tablet copy: Starting download of 0 data blocks...
I20260926 13:40:21.713651 7907 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: tablet copy: Starting download of 1 WAL segments...
I20260926 13:40:21.715420 7907 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 13:40:21.717502 7355 tablet_copy_service.cc:342] P ca6d743256e747e7bd97ca28676150d4: Request end of tablet copy session 753d614d1cb643beaa988ad729abba2e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.49.60@KRBTEST.COM'} at 127.0.0.1:59886
I20260926 13:40:21.717614 7355 tablet_copy_service.cc:433] P ca6d743256e747e7bd97ca28676150d4: ending tablet copy session 753d614d1cb643beaa988ad729abba2e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 753d614d1cb643beaa988ad729abba2e
I20260926 13:40:21.726028 7907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:21.728888 8095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:21.729048 8093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:21.729290 7907 server_base.cc:1034] running on GCE node
W20260926 13:40:21.732004 8092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:21.732405 7907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:21.732829 7907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:21.734045 7907 hybrid_clock.cc:648] HybridClock initialized: now 1790430021734022 us; error 51 us; skew 500 ppm
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430021, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:21.736012 7907 init.cc:377] Logged in from keytab as oryx/127.1.49.60@KRBTEST.COM (short username oryx)
I20260926 13:40:21.736656 7907 webserver.cc:460] Webserver started at http://127.1.49.60:40387/ using document root <none> and password file <none>
I20260926 13:40:21.736951 7907 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:21.737010 7907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:21.738898 7907 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:40:21.740346 8102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:21.740726 7907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:21.740818 7907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/wal
uuid: "753d614d1cb643beaa988ad729abba2e"
format_stamp: "Formatted at 2026-09-26 13:40:21 on dist-test-slave-1wns"
server_key: "a1c026074f32b0360cb7d85d3612c384"
server_key_iv: "55b37ae01bf6914ad7fed4721afa6722"
server_key_version: "encryptionkey@0"
I20260926 13:40:21.740940 7907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:21.766368 7907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:21.768821 7907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:21.774367 7907 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.60:34631
I20260926 13:40:21.774401 8154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.60:34631 every 8 connection(s)
I20260926 13:40:21.775049 7907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/master-2/data/info.pb
I20260926 13:40:21.777343 8155 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:40:21.778700 8155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: Bootstrap starting.
I20260926 13:40:21.791417 8155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:40:21.791786 8155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: Bootstrap complete.
I20260926 13:40:21.792483 8155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } }
I20260926 13:40:21.792639 8155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 753d614d1cb643beaa988ad729abba2e, State: Initialized, Role: LEARNER
W20260926 13:40:21.792393 8080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:21.792794 8080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260926 13:40:21.792753 8155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } }
W20260926 13:40:21.792882 8080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 13:40:21.793150 8161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } } }
I20260926 13:40:21.793292 8161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [sys.catalog]: This master's current role is: LEARNER
I20260926 13:40:21.793292 8155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:21.796569 8172 catalog_manager.cc:1259] Loaded cluster ID: f166fbec81484299be1710d5dd29f21d
I20260926 13:40:21.796643 8172 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: loading cluster ID for follower catalog manager: success
I20260926 13:40:21.798347 8172 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: acquiring CA information for follower catalog manager: success
W20260926 13:40:21.799306 8080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:21.799404 8080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:21.799542 8080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:40:21.799649 8172 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:40:21.805931 8080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.1.keytab
--principal=oryx/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:45087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=43021
--tserver_master_addrs=127.1.49.62:36609,127.1.49.61:34937,127.1.49.60:34631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:21.807231 8080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:21.808506 8080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:21.819455 8177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:21.819489 8174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:21.819483 8175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:21.819875 8080 server_base.cc:1034] running on GCE node
I20260926 13:40:21.820405 8080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:21.821089 8080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:21.822273 8080 hybrid_clock.cc:648] HybridClock initialized: now 1790430021822254 us; error 49 us; skew 500 ppm
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430021, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:21.825477 8080 init.cc:377] Logged in from keytab as oryx/127.1.49.1@KRBTEST.COM (short username oryx)
I20260926 13:40:21.826589 8080 webserver.cc:460] Webserver started at http://127.1.49.1:43021/ using document root <none> and password file <none>
I20260926 13:40:21.827196 8080 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:21.827308 8080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:21.831532 8080 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:21.834012 8184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:21.835366 8080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 13:40:21.835584 8080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "04dccfca386341c08195a928a2aecd0e"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "de16939b2670457a6a551ad27d776b73"
server_key_iv: "2a960f91e0e0b75ad099bf40e863ac8f"
server_key_version: "encryptionkey@0"
I20260926 13:40:21.836236 8080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:21.848876 8080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:21.852547 8080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:21.852767 8080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:21.853411 8080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:21.854301 8080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:21.854369 8080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:21.854442 8080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:21.854492 8080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:21.866104 8080 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:45087
I20260926 13:40:21.866137 8297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:45087 every 8 connection(s)
I20260926 13:40:21.867606 8080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:40:21.877586 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 8080
I20260926 13:40:21.877825 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 7531
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430021, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.60@KRBTEST.COM
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430021, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.61@KRBTEST.COM
Sep 26 13:40:21 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430021, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
I20260926 13:40:21.887048 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:34249
--local_ip_for_outbound_sockets=127.1.49.2
--tserver_master_addrs=127.1.49.62:36609
--webserver_port=38575
--webserver_interface=127.1.49.2
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.2.keytab
--principal=oryx/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.49.62:36609,127.1.49.61:34937
--tserver_master_addrs=127.1.49.62:36609,127.1.49.61:34937,127.1.49.60:34631 with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:40:21.889855 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.872091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:50699 (local address 127.1.49.62:36609)
0926 13:40:21.872229 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.872234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.872271 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:40:21.873232 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.873236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.873309 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.873368 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.874969 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.875563 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.877549 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.877971 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.878079 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.885829 (+ 7750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.885837 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.885844 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.885885 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.887791 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.888374 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.888378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.888380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.888470 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.888850 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.888854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.888856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.889082 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.889179 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.889518 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.889664 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:40:21.889855 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.872454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:33785 (local address 127.1.49.61:34937)
0926 13:40:21.872583 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.872588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.872910 (+ 322us) server_negotiation.cc:407] Connection header received
0926 13:40:21.873209 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.873213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.873274 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.873349 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.875929 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.876450 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.877758 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.878133 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.878213 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.885143 (+ 6930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.885147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.885150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.885185 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.886888 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.887616 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.887623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.887628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.887737 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.888340 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.888348 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.888351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.888642 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.888722 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.889518 (+ 796us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.889664 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:40:21.890744 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.870723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:44117 (local address 127.1.49.60:34631)
0926 13:40:21.871056 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:21.871073 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:21.872558 (+ 1485us) server_negotiation.cc:407] Connection header received
0926 13:40:21.873321 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:21.873324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:21.873385 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:21.873478 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:21.875763 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.876976 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.878791 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.879173 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.879297 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.883109 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:21.883112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:21.883114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:21.883159 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:21.884961 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.887650 (+ 2689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.887654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.887656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.887767 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:21.888295 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:21.888299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:21.888301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:21.888590 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:21.888677 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:21.890393 (+ 1716us) server_negotiation.cc:299] Negotiation successful
0926 13:40:21.890551 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":91,"threads_started":1}
I20260926 13:40:21.891501 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.870458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:34631 (local address 127.1.49.1:44117)
0926 13:40:21.872423 (+ 1965us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.872437 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.872994 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.873542 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.873552 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.874110 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.874796 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.874815 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.877116 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.877120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.877929 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.877937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.879299 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.879303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.879413 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.879968 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.879979 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.882854 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.885480 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.885489 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.885513 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.887506 (+ 1993us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.887920 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.887926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.887929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.888166 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.888749 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.888757 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.889205 (+ 448us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.889440 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.889867 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1868,"mutex_wait_us":1646,"spinlock_wait_cycles":23040,"thread_start_us":132,"threads_started":1}
I20260926 13:40:21.891510 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.870490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.1:50699)
0926 13:40:21.871670 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.871741 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.872994 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.873546 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.873554 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.874107 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.874795 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.874812 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.875726 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.875731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.877380 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.877393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.880134 (+ 2741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.880141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.880302 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.883145 (+ 2843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.883165 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.885617 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.887974 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.887978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.887980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.888244 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.888624 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.888628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.888630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.888722 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.889241 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.889244 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.889329 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.890087 (+ 758us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.890274 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":128,"thread_start_us":712,"threads_started":1}
I20260926 13:40:21.891510 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:21.872365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:34937 (local address 127.1.49.1:33785)
0926 13:40:21.872815 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:21.872829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:21.872997 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:21.873542 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:21.873552 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:21.874113 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:21.874801 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.874815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.876567 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.876578 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.877628 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:21.877638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:21.878427 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:21.878433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:21.878777 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:21.879876 (+ 1099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:21.879901 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:21.884952 (+ 5051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:21.887032 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.887036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.887038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.887447 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.887897 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:21.887902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:21.887908 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:21.888182 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:21.888802 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:21.888806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:21.889205 (+ 399us) client_negotiation.cc:769] Sending connection context
0926 13:40:21.890023 (+ 818us) client_negotiation.cc:240] Negotiation successful
0926 13:40:21.890210 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":190,"spinlock_wait_cycles":66944,"thread_start_us":89,"threads_started":1}
I20260926 13:40:21.894697 8299 heartbeater.cc:347] Connected to a master server at 127.1.49.62:36609
I20260926 13:40:21.894668 8300 heartbeater.cc:347] Connected to a master server at 127.1.49.61:34937
I20260926 13:40:21.894675 8298 heartbeater.cc:347] Connected to a master server at 127.1.49.60:34631
I20260926 13:40:21.894994 8300 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:21.894992 8299 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:21.895072 8298 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:21.895677 8298 heartbeater.cc:511] Master 127.1.49.60:34631 requested a full tablet report, sending...
I20260926 13:40:21.896000 8299 heartbeater.cc:511] Master 127.1.49.62:36609 requested a full tablet report, sending...
I20260926 13:40:21.896000 8300 heartbeater.cc:511] Master 127.1.49.61:34937 requested a full tablet report, sending...
I20260926 13:40:21.897217 7335 ts_manager.cc:194] Re-registered known tserver with Master: 04dccfca386341c08195a928a2aecd0e (127.1.49.1:45087)
I20260926 13:40:21.897506 7928 ts_manager.cc:194] Registered new tserver with Master: 04dccfca386341c08195a928a2aecd0e (127.1.49.1:45087)
I20260926 13:40:21.897661 8119 ts_manager.cc:194] Registered new tserver with Master: 04dccfca386341c08195a928a2aecd0e (127.1.49.1:45087)
I20260926 13:40:21.898058 7335 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:50699
I20260926 13:40:21.933034 7794 heartbeater.cc:503] Master 127.1.49.62:36609 was elected leader, sending a full tablet report...
W20260926 13:40:22.016595 8306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:22.016924 8306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:22.017020 8306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:22.021775 8306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:22.021878 8306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:22.021976 8306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:40:22.027748 8306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.2.keytab
--principal=oryx/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:34249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=38575
--tserver_master_addrs=127.1.49.62:36609,127.1.49.61:34937,127.1.49.60:34631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:22.029062 8306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:22.030248 8306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:22.041380 8312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:22.041399 8314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:22.041512 8311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:22.041754 8306 server_base.cc:1034] running on GCE node
I20260926 13:40:22.042301 8306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:22.043076 8306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:22.044294 8306 hybrid_clock.cc:648] HybridClock initialized: now 1790430022044253 us; error 64 us; skew 500 ppm
Sep 26 13:40:22 dist-test-slave-1wns krb5kdc[7290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430022, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:22.048098 8306 init.cc:377] Logged in from keytab as oryx/127.1.49.2@KRBTEST.COM (short username oryx)
I20260926 13:40:22.049338 8306 webserver.cc:460] Webserver started at http://127.1.49.2:38575/ using document root <none> and password file <none>
I20260926 13:40:22.050034 8306 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:22.050235 8306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:22.055253 8306 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:40:22.058086 8321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:22.060005 8306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:22.060196 8306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "3e0ce36f39e9402883d8942956b16c27"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "f2cb6b518a9bcb7cdd51f7d44ff76df8"
server_key_iv: "13f610e109ec18cb05bccfeaa3d3be93"
server_key_version: "encryptionkey@0"
I20260926 13:40:22.060920 8306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:22.075888 8306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:22.079746 8306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:22.079990 8306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:22.080636 8306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:22.081549 8306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:22.081602 8306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:22.081643 8306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:22.081696 8306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:22.091737 8306 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:34249
I20260926 13:40:22.091792 8434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:34249 every 8 connection(s)
I20260926 13:40:22.092877 8306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:40:22.100064 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 8306
I20260926 13:40:22.100291 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 7665
Sep 26 13:40:22 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430022, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.60@KRBTEST.COM
Sep 26 13:40:22 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430022, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.61@KRBTEST.COM
I20260926 13:40:22.109251 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:42317
--local_ip_for_outbound_sockets=127.1.49.3
--tserver_master_addrs=127.1.49.62:36609
--webserver_port=38883
--webserver_interface=127.1.49.3
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.3.keytab
--principal=oryx/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.49.62:36609,127.1.49.61:34937
--tserver_master_addrs=127.1.49.62:36609,127.1.49.61:34937,127.1.49.60:34631 with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:40:22 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430022, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
I20260926 13:40:22.114033 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.094904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:33409 (local address 127.1.49.60:34631)
0926 13:40:22.095121 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.095125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.096502 (+ 1377us) server_negotiation.cc:407] Connection header received
0926 13:40:22.097799 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.097803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.097870 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.097969 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:22.099922 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.100804 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.103436 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.103874 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.103954 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.108191 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:22.108196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:22.108199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:22.108240 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:22.110777 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.111697 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.111701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.111703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.111832 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.112358 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.112363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.112364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.112597 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:22.112687 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.113592 (+ 905us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.113764 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260926 13:40:22.114984 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.097138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:55649 (local address 127.1.49.61:34937)
0926 13:40:22.097261 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.097266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.098553 (+ 1287us) server_negotiation.cc:407] Connection header received
0926 13:40:22.098738 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.098742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.098799 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.098898 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:22.099904 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.100868 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.103221 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.103709 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.103871 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.108411 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:22.108416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:22.108418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:22.108456 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:22.110723 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.111696 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.111700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.111702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.111832 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.112348 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.112352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.112354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.112609 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:22.112686 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.114680 (+ 1994us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.114819 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:40:22.115252 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.095553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:34631 (local address 127.1.49.2:33409)
0926 13:40:22.096312 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:22.096375 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:22.097546 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:22.098236 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:22.098248 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:22.098863 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:22.099652 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.099674 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.102234 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.102238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.103284 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.103295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.104149 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.104153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.104456 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.105395 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:22.105411 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:22.107894 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:22.110899 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:22.110907 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:22.110931 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:22.111552 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:22.111964 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:22.111967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:22.111970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:22.112218 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:22.112767 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:22.112775 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:22.113220 (+ 445us) client_negotiation.cc:769] Sending connection context
0926 13:40:22.113487 (+ 267us) client_negotiation.cc:240] Negotiation successful
0926 13:40:22.113925 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":72,"spinlock_wait_cycles":22656,"thread_start_us":769,"threads_started":1}
I20260926 13:40:22.116789 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.094985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:43125 (local address 127.1.49.62:36609)
0926 13:40:22.095120 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.095125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.098141 (+ 3016us) server_negotiation.cc:407] Connection header received
0926 13:40:22.098318 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.098321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.098382 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.098466 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:22.106683 (+ 8217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.107656 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.109138 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.109533 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.109635 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.112698 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:22.112703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:22.112706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:22.112743 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:22.114636 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.115196 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.115203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.115205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.115296 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.115662 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.115666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.115668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.115968 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:22.116062 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.116491 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.116628 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:40:22.117165 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.095588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.2:43125)
0926 13:40:22.098029 (+ 2441us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:22.098045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:22.098178 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:22.098611 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:22.098614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:22.099220 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:22.099654 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.099676 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.107876 (+ 8200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.107884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.108974 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.108989 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.109870 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.109877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.110021 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.110590 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:22.110612 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:22.112448 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:22.114765 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:22.114771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:22.114774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:22.115016 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:22.115424 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:22.115429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:22.115432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:22.115545 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:22.116158 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:22.116161 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:22.116273 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 13:40:22.116415 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 13:40:22.116577 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2373,"mutex_wait_us":198,"spinlock_wait_cycles":5504,"thread_start_us":290,"threads_started":1}
I20260926 13:40:22.117313 8435 heartbeater.cc:347] Connected to a master server at 127.1.49.60:34631
I20260926 13:40:22.117580 8435 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:22.117851 8436 heartbeater.cc:347] Connected to a master server at 127.1.49.62:36609
I20260926 13:40:22.117975 8436 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:22.118228 8435 heartbeater.cc:511] Master 127.1.49.60:34631 requested a full tablet report, sending...
I20260926 13:40:22.118234 8436 heartbeater.cc:511] Master 127.1.49.62:36609 requested a full tablet report, sending...
I20260926 13:40:22.119534 7335 ts_manager.cc:194] Re-registered known tserver with Master: 3e0ce36f39e9402883d8942956b16c27 (127.1.49.2:34249)
I20260926 13:40:22.120303 8119 ts_manager.cc:194] Registered new tserver with Master: 3e0ce36f39e9402883d8942956b16c27 (127.1.49.2:34249)
I20260926 13:40:22.120415 7335 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:43125
I20260926 13:40:22.124066 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.097017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:34937 (local address 127.1.49.2:55649)
0926 13:40:22.098464 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:22.098479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:22.098621 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:22.098912 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:22.098916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:22.099215 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:22.099654 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.099676 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.101165 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.101173 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.103056 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.103072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.104149 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.104157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.104467 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.105323 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:22.105353 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:22.108207 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:22.110899 (+ 2692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:22.110907 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:22.110931 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:22.111552 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:22.111964 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:22.111967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:22.111970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:22.112218 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:22.112767 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:22.112775 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:22.114459 (+ 1684us) client_negotiation.cc:769] Sending connection context
0926 13:40:22.115240 (+ 781us) client_negotiation.cc:240] Negotiation successful
0926 13:40:22.115408 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"mutex_wait_us":1640,"spinlock_wait_cycles":12672,"thread_start_us":102,"threads_started":1}
I20260926 13:40:22.124795 8437 heartbeater.cc:347] Connected to a master server at 127.1.49.61:34937
I20260926 13:40:22.124915 8437 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:22.125164 8437 heartbeater.cc:511] Master 127.1.49.61:34937 requested a full tablet report, sending...
I20260926 13:40:22.125957 7928 ts_manager.cc:194] Registered new tserver with Master: 3e0ce36f39e9402883d8942956b16c27 (127.1.49.2:34249)
W20260926 13:40:22.249323 8442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:22.249672 8442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:22.249722 8442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:22.254537 8442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:22.254617 8442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:22.254681 8442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:40:22.260646 8442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.3.keytab
--principal=oryx/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:42317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=38883
--tserver_master_addrs=127.1.49.62:36609,127.1.49.61:34937,127.1.49.60:34631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:22.261762 8442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:22.262786 8442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:22.273429 8449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:22.273414 8448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:22.273414 8451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:22.273738 8442 server_base.cc:1034] running on GCE node
I20260926 13:40:22.274253 8442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:22.275027 8442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:22.276222 8442 hybrid_clock.cc:648] HybridClock initialized: now 1790430022276210 us; error 36 us; skew 500 ppm
Sep 26 13:40:22 dist-test-slave-1wns krb5kdc[7290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430022, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:22.279469 8442 init.cc:377] Logged in from keytab as oryx/127.1.49.3@KRBTEST.COM (short username oryx)
I20260926 13:40:22.280509 8442 webserver.cc:460] Webserver started at http://127.1.49.3:38883/ using document root <none> and password file <none>
I20260926 13:40:22.281102 8442 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:22.281216 8442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:22.285666 8442 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:40:22.288208 8458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:22.289675 8442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 13:40:22.289835 8442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "33d6d78df9f445f5852b6f10003d12da"
format_stamp: "Formatted at 2026-09-26 13:40:20 on dist-test-slave-1wns"
server_key: "c2798bfd3399944b949c2d67768ef0ee"
server_key_iv: "d37ff1371b46f6417eb977f1f391c34f"
server_key_version: "encryptionkey@0"
I20260926 13:40:22.290515 8442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:22.298197 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.294175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52976 (local address 127.1.49.60:34631)
0926 13:40:22.294326 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.294331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.294668 (+ 337us) server_negotiation.cc:407] Connection header received
0926 13:40:22.294884 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.294888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.294948 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.295067 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:22.295340 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.295962 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.297186 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.297643 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.297769 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.297939 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.297988 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.298057 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:40:22.298197 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.294111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:34631 (local address 127.0.0.1:52976)
0926 13:40:22.294547 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:22.294560 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:22.294735 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:22.295065 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:22.295068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:22.295087 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:22.295220 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.295226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.296078 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.296082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.297053 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.297064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.297780 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.297783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.297876 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.297902 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 13:40:22.297967 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:40:22.298053 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":75,"threads_started":1}
I20260926 13:40:22.298818 8129 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 13:40:22.299321 8464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca6d743256e747e7bd97ca28676150d4. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } } }
I20260926 13:40:22.299480 8464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [sys.catalog]: This master's current role is: LEARNER
I20260926 13:40:22.315982 8442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:22.320432 8442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:22.320680 8442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:22.321448 8442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:22.322453 8442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:22.322531 8442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:22.322611 8442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:22.322644 8442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:22.333098 8442 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:42317
I20260926 13:40:22.333139 8573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:42317 every 8 connection(s)
I20260926 13:40:22.334227 8442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:40:22.340355 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 8442
Sep 26 13:40:22 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430022, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
Sep 26 13:40:22 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430022, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.60@KRBTEST.COM
Sep 26 13:40:22 dist-test-slave-1wns krb5kdc[7290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430022, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.61@KRBTEST.COM
I20260926 13:40:22.355650 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.336404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:52025 (local address 127.1.49.62:36609)
0926 13:40:22.336756 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.336761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.337751 (+ 990us) server_negotiation.cc:407] Connection header received
0926 13:40:22.338810 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.338814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.338878 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.339021 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:22.341083 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.342007 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.344066 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.344458 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.344544 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.348852 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:22.348859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:22.348878 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:22.348912 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:22.353153 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.353721 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.353725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.353727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.353823 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.354123 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.354126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.354129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.354359 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:22.354465 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.355259 (+ 794us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.355444 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260926 13:40:22.356082 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.336242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:34303 (local address 127.1.49.60:34631)
0926 13:40:22.336413 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.336417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.337529 (+ 1112us) server_negotiation.cc:407] Connection header received
0926 13:40:22.338937 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.338939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.338990 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.339082 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:22.341062 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.341873 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.344043 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.344404 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.344490 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.350210 (+ 5720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:22.350215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:22.350218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:22.350248 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:22.352304 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.353085 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.353093 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.353097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.353192 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.353669 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.353673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.353676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.353918 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:22.354019 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.355738 (+ 1719us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.355904 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 13:40:22.356307 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.337956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:45897 (local address 127.1.49.61:34937)
0926 13:40:22.338090 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.338094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.338271 (+ 177us) server_negotiation.cc:407] Connection header received
0926 13:40:22.339203 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.339206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.339255 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.339335 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:22.341236 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.342114 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.344066 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.344464 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.344544 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.350180 (+ 5636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:22.350184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:22.350187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:22.350216 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:22.353920 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.354468 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.354474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.354477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.354568 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.354880 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.354884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.354886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.355144 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:22.355699 (+ 555us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.356047 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.356164 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:40:22.357757 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.336690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:34631 (local address 127.1.49.3:34303)
0926 13:40:22.337326 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:22.337406 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:22.338582 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:22.339215 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:22.339224 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:22.339913 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:22.340829 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.340851 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.342094 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.342097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.343896 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.343910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.344707 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.344713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.345018 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.345992 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:22.346003 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:22.350025 (+ 4022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:22.352475 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:22.352486 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:22.352514 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:22.352936 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:22.353306 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:22.353308 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:22.353310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:22.353560 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:22.355413 (+ 1853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:22.355419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:22.355527 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 13:40:22.356270 (+ 743us) client_negotiation.cc:240] Negotiation successful
0926 13:40:22.356802 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":1259,"spinlock_wait_cycles":21504,"thread_start_us":701,"threads_started":1}
I20260926 13:40:22.358453 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.337797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:34937 (local address 127.1.49.3:45897)
0926 13:40:22.338170 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:22.338184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:22.338585 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:22.339375 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:22.339378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:22.339944 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:22.341102 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.341110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.342293 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.342296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.343910 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.343921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.344716 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.344721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.345017 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.345950 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:22.345973 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:22.349601 (+ 3628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:22.354046 (+ 4445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:22.354052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:22.354056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:22.354319 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:22.354686 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:22.354690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:22.354693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:22.354776 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:22.355754 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:22.355756 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:22.355866 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 13:40:22.356435 (+ 569us) client_negotiation.cc:240] Negotiation successful
0926 13:40:22.356555 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":770,"spinlock_wait_cycles":5632,"thread_start_us":86,"threads_started":1}
I20260926 13:40:22.358721 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.336691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.3:52025)
0926 13:40:22.337636 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:22.337654 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:22.338581 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:22.339215 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:22.339223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:22.339946 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:22.340829 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.340851 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.342156 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.342161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.343910 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.343921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.344730 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.344734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.345508 (+ 774us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.346508 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:22.346526 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:22.348636 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:22.353303 (+ 4667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:22.353306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:22.353308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:22.353588 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:22.353929 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:22.353933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:22.353935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:22.354022 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:22.354534 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:22.354542 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:22.354928 (+ 386us) client_negotiation.cc:769] Sending connection context
0926 13:40:22.355890 (+ 962us) client_negotiation.cc:240] Negotiation successful
0926 13:40:22.356383 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":928,"spinlock_wait_cycles":36736,"thread_start_us":144,"threads_started":1}
I20260926 13:40:22.359656 8576 heartbeater.cc:347] Connected to a master server at 127.1.49.61:34937
I20260926 13:40:22.359768 8574 heartbeater.cc:347] Connected to a master server at 127.1.49.60:34631
I20260926 13:40:22.359963 8576 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:22.359972 8574 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:22.360642 8574 heartbeater.cc:511] Master 127.1.49.60:34631 requested a full tablet report, sending...
I20260926 13:40:22.360746 8575 heartbeater.cc:347] Connected to a master server at 127.1.49.62:36609
I20260926 13:40:22.360813 8575 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:22.360997 8575 heartbeater.cc:511] Master 127.1.49.62:36609 requested a full tablet report, sending...
I20260926 13:40:22.361225 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.343873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.1.49.60:34631)
0926 13:40:22.344399 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.344405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.349207 (+ 4802us) server_negotiation.cc:407] Connection header received
0926 13:40:22.349258 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.349261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.349322 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.349410 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:22.350378 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.351221 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.352452 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.352854 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.352993 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.355074 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:22.355077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:22.355128 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:22.355152 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:22.359144 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.359604 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.359608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.359610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.359715 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.360050 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.360054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.360056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.360290 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:22.360408 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.360931 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.361058 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":83,"threads_started":1}
I20260926 13:40:22.361224 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.343127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49184 (local address 127.1.49.61:34937)
0926 13:40:22.345114 (+ 1987us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.345118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.345718 (+ 600us) server_negotiation.cc:407] Connection header received
0926 13:40:22.345770 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.345774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.345829 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.345929 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:22.347031 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.347672 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.348914 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.349278 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.349399 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.351134 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:22.351138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:22.351140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:22.351164 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:22.358222 (+ 7058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.359267 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.359272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.359277 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.359358 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.359848 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.359851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.359853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.360096 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:22.360220 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.360934 (+ 714us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.361067 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1911,"thread_start_us":85,"threads_started":1}
I20260926 13:40:22.361987 8118 ts_manager.cc:194] Registered new tserver with Master: 33d6d78df9f445f5852b6f10003d12da (127.1.49.3:42317)
I20260926 13:40:22.362726 8576 heartbeater.cc:511] Master 127.1.49.61:34937 requested a full tablet report, sending...
I20260926 13:40:22.363507 7335 ts_manager.cc:194] Re-registered known tserver with Master: 33d6d78df9f445f5852b6f10003d12da (127.1.49.3:42317)
I20260926 13:40:22.363593 7928 ts_manager.cc:194] Registered new tserver with Master: 33d6d78df9f445f5852b6f10003d12da (127.1.49.3:42317)
I20260926 13:40:22.364277 7335 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:52025
I20260926 13:40:22.366699 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.342746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.1.49.62:36609)
0926 13:40:22.345076 (+ 2330us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.345080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.345095 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:40:22.345139 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.345144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.345199 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.345294 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:22.347157 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.348364 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.354382 (+ 6018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.354735 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.354831 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.357428 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:22.357432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:22.357435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:22.357462 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:22.364524 (+ 7062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.365064 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.365068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.365070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.365171 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.365607 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.365611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.365613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.365839 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:22.365988 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.366361 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.366516 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2247,"thread_start_us":68,"threads_started":1}
I20260926 13:40:22.369681 7335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:40:22.373034 7335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:40:22.376471 8446 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: NON_VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: true } } }
I20260926 13:40:22.376792 8446 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:22.381685 8446 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4: attempting to promote NON_VOTER 753d614d1cb643beaa988ad729abba2e to VOTER
I20260926 13:40:22.382403 8446 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } }
I20260926 13:40:22.384701 7944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 FOLLOWER]: Refusing update from remote peer ca6d743256e747e7bd97ca28676150d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 13:40:22.385406 8593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 13:40:22.387498 8129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 1 LEARNER]: Refusing update from remote peer ca6d743256e747e7bd97ca28676150d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 13:40:22.388152 8593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 13:40:22.390064 7944 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 753d614d1cb643beaa988ad729abba2e (127.1.49.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } } }
I20260926 13:40:22.391480 8129 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 753d614d1cb643beaa988ad729abba2e (127.1.49.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } } }
I20260926 13:40:22.392513 8602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } } }
I20260926 13:40:22.392704 8602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:40:22.393692 8464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } } }
I20260926 13:40:22.393913 8464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:40:22.388054 8592 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 753d614d1cb643beaa988ad729abba2e (127.1.49.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } } }
I20260926 13:40:22.397078 8593 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } } }
I20260926 13:40:22.397238 8593 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:22.397531 8593 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } } }
I20260926 13:40:22.397655 8593 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:22.395879 8592 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ca6d743256e747e7bd97ca28676150d4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } } }
I20260926 13:40:22.398573 8592 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:22.405570 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.399643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:45087 (local address 127.0.0.1:57148)
0926 13:40:22.399878 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:22.399895 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:22.399986 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:22.400870 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:22.400873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:22.400909 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:22.401024 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.401032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.402610 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.402613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.403756 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.403765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.405161 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.405164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.405252 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.405278 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 13:40:22.405333 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 13:40:22.405405 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 13:40:22.406509 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.399856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57148 (local address 127.1.49.1:45087)
0926 13:40:22.400174 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.400210 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.400239 (+ 29us) server_negotiation.cc:407] Connection header received
0926 13:40:22.400331 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.400340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.400699 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.400888 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:22.401139 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.402473 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.404201 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.405027 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.406046 (+ 1019us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.406209 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.406272 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.406379 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
I20260926 13:40:22.408349 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.400073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:42317 (local address 127.0.0.1:37032)
0926 13:40:22.400333 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:22.400347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:22.400431 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:22.402646 (+ 2215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:22.402649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:22.402670 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:22.402775 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.402783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.404501 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.404504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.405928 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.405940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.407775 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.407780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.407929 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.407963 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 13:40:22.408052 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 13:40:22.408148 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":41,"thread_start_us":93,"threads_started":1}
I20260926 13:40:22.409021 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.400372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:34249 (local address 127.0.0.1:44152)
0926 13:40:22.401541 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:22.401555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:22.401649 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:22.403315 (+ 1666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:22.403319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:22.403335 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:22.403414 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.403419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.405500 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.405503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.406615 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.406627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.408648 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.408651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.408744 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.408767 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:40:22.408828 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 13:40:22.408894 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260926 13:40:22.409228 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.400353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.1.49.3:42317)
0926 13:40:22.401965 (+ 1612us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.401979 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.402001 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:40:22.402098 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.402104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.402365 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.402557 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:22.402923 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.404331 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.406078 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.407371 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.407514 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.407698 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.408887 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.409022 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1485,"thread_start_us":118,"threads_started":1}
I20260926 13:40:22.409974 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.400790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44152 (local address 127.1.49.2:34249)
0926 13:40:22.402873 (+ 2083us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.402883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.402905 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:40:22.402978 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.402983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.403166 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.403702 (+ 536us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:22.403882 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.405388 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.406874 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.408502 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.409538 (+ 1036us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.409687 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.409740 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.409844 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1970,"thread_start_us":100,"threads_started":1}
I20260926 13:40:22.410043 8232 tablet_service.cc:1467] Processing CreateTablet for tablet e6c2c1654fec44cfa04d0c9dffb57e12 (DEFAULT_TABLE table=test-table [id=a9f0f1864b7844db871582b38d7524bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:22.411270 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c2c1654fec44cfa04d0c9dffb57e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:22.412846 8367 tablet_service.cc:1467] Processing CreateTablet for tablet e6c2c1654fec44cfa04d0c9dffb57e12 (DEFAULT_TABLE table=test-table [id=a9f0f1864b7844db871582b38d7524bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:22.413009 8508 tablet_service.cc:1467] Processing CreateTablet for tablet e6c2c1654fec44cfa04d0c9dffb57e12 (DEFAULT_TABLE table=test-table [id=a9f0f1864b7844db871582b38d7524bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:22.413941 8367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c2c1654fec44cfa04d0c9dffb57e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:22.414124 8508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c2c1654fec44cfa04d0c9dffb57e12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:22.420161 8608 tablet_bootstrap.cc:492] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e: Bootstrap starting.
I20260926 13:40:22.423305 8608 tablet_bootstrap.cc:654] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:22.423748 8609 tablet_bootstrap.cc:492] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da: Bootstrap starting.
I20260926 13:40:22.424331 8608 log.cc:824] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:22.426110 8611 tablet_bootstrap.cc:492] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27: Bootstrap starting.
I20260926 13:40:22.426523 8609 tablet_bootstrap.cc:654] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:22.426689 8608 tablet_bootstrap.cc:492] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e: No bootstrap required, opened a new log
I20260926 13:40:22.426898 8608 ts_tablet_manager.cc:1397] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 13:40:22.427492 8609 log.cc:824] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:22.428666 8611 tablet_bootstrap.cc:654] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:22.429579 8611 log.cc:824] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:22.429792 8609 tablet_bootstrap.cc:492] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da: No bootstrap required, opened a new log
I20260926 13:40:22.430039 8609 ts_tablet_manager.cc:1397] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 13:40:22.431528 8608 raft_consensus.cc:348] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } } peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } }
I20260926 13:40:22.431859 8611 tablet_bootstrap.cc:492] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27: No bootstrap required, opened a new log
I20260926 13:40:22.432065 8611 ts_tablet_manager.cc:1397] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:40:22.432377 8608 raft_consensus.cc:374] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:22.432452 8608 raft_consensus.cc:729] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04dccfca386341c08195a928a2aecd0e, State: Initialized, Role: FOLLOWER
I20260926 13:40:22.433040 8608 consensus_queue.cc:260] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } } peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } }
I20260926 13:40:22.434216 8608 ts_tablet_manager.cc:1428] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260926 13:40:22.435693 8609 raft_consensus.cc:348] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } } peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } }
I20260926 13:40:22.436034 8609 raft_consensus.cc:374] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:22.436094 8609 raft_consensus.cc:729] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d6d78df9f445f5852b6f10003d12da, State: Initialized, Role: FOLLOWER
I20260926 13:40:22.436642 8609 consensus_queue.cc:260] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } } peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } }
I20260926 13:40:22.437098 8611 raft_consensus.cc:348] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } } peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } }
I20260926 13:40:22.437372 8611 raft_consensus.cc:374] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:22.437430 8611 raft_consensus.cc:729] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0ce36f39e9402883d8942956b16c27, State: Initialized, Role: FOLLOWER
I20260926 13:40:22.437649 8299 heartbeater.cc:503] Master 127.1.49.62:36609 was elected leader, sending a full tablet report...
I20260926 13:40:22.437865 8609 ts_tablet_manager.cc:1428] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260926 13:40:22.437876 8611 consensus_queue.cc:260] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } } peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } }
I20260926 13:40:22.438939 8611 ts_tablet_manager.cc:1428] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 13:40:22.439234 8436 heartbeater.cc:503] Master 127.1.49.62:36609 was elected leader, sending a full tablet report...
I20260926 13:40:22.440411 8575 heartbeater.cc:503] Master 127.1.49.62:36609 was elected leader, sending a full tablet report...
I20260926 13:40:22.557755 8616 raft_consensus.cc:482] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:40:22.558001 8616 raft_consensus.cc:504] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } } peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } }
I20260926 13:40:22.559226 8616 leader_election.cc:288] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33d6d78df9f445f5852b6f10003d12da (127.1.49.3:42317), 04dccfca386341c08195a928a2aecd0e (127.1.49.1:45087)
I20260926 13:40:22.565052 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.559594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:32817 (local address 127.1.49.3:42317)
0926 13:40:22.559727 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.559732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.559789 (+ 57us) server_negotiation.cc:407] Connection header received
0926 13:40:22.560009 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.560012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.560068 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.560153 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:22.560701 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.561492 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.563536 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.564187 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.564301 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.564371 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.564857 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.564936 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:40:22.565074 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.559503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:42317 (local address 127.1.49.2:32817)
0926 13:40:22.559718 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:22.559737 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:22.559870 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:22.560190 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:22.560193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:22.560227 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:22.560573 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.560580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.561656 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.561660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.563412 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.563424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.564324 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.564328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.564583 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.564691 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 13:40:22.564785 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 13:40:22.564909 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":236,"spinlock_wait_cycles":8576}
I20260926 13:40:22.565121 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.559603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:47987 (local address 127.1.49.1:45087)
0926 13:40:22.559741 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.559747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.559957 (+ 210us) server_negotiation.cc:407] Connection header received
0926 13:40:22.560032 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.560035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.560082 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.560154 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:22.560701 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.561516 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.563494 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.564187 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.564301 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.564371 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.564908 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.565002 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:40:22.565418 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.559503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:45087 (local address 127.1.49.2:47987)
0926 13:40:22.559839 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:22.559861 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:22.559940 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:22.560190 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:22.560193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:22.560225 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:22.560571 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.560580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.561656 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.561660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.563320 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:22.563335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.564324 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.564328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:22.564585 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.564693 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 13:40:22.564785 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 13:40:22.565306 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":854,"spinlock_wait_cycles":7680}
I20260926 13:40:22.566046 8528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c2c1654fec44cfa04d0c9dffb57e12" candidate_uuid: "3e0ce36f39e9402883d8942956b16c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d6d78df9f445f5852b6f10003d12da" is_pre_election: true
I20260926 13:40:22.566058 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c2c1654fec44cfa04d0c9dffb57e12" candidate_uuid: "3e0ce36f39e9402883d8942956b16c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04dccfca386341c08195a928a2aecd0e" is_pre_election: true
I20260926 13:40:22.566357 8528 raft_consensus.cc:2454] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e0ce36f39e9402883d8942956b16c27 in term 0.
I20260926 13:40:22.566388 8252 raft_consensus.cc:2454] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e0ce36f39e9402883d8942956b16c27 in term 0.
I20260926 13:40:22.566898 8322 leader_election.cc:302] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04dccfca386341c08195a928a2aecd0e, 33d6d78df9f445f5852b6f10003d12da, 3e0ce36f39e9402883d8942956b16c27; no voters:
I20260926 13:40:22.567237 8616 raft_consensus.cc:2781] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:40:22.567317 8616 raft_consensus.cc:482] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:40:22.567348 8616 raft_consensus.cc:3037] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:22.568598 8616 raft_consensus.cc:504] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } } peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } }
I20260926 13:40:22.569022 8616 leader_election.cc:288] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [CANDIDATE]: Term 1 election: Requested vote from peers 33d6d78df9f445f5852b6f10003d12da (127.1.49.3:42317), 04dccfca386341c08195a928a2aecd0e (127.1.49.1:45087)
I20260926 13:40:22.569496 8528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c2c1654fec44cfa04d0c9dffb57e12" candidate_uuid: "3e0ce36f39e9402883d8942956b16c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d6d78df9f445f5852b6f10003d12da"
I20260926 13:40:22.569593 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c2c1654fec44cfa04d0c9dffb57e12" candidate_uuid: "3e0ce36f39e9402883d8942956b16c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04dccfca386341c08195a928a2aecd0e"
I20260926 13:40:22.569612 8528 raft_consensus.cc:3037] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:22.569778 8252 raft_consensus.cc:3037] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:22.570730 8528 raft_consensus.cc:2454] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e0ce36f39e9402883d8942956b16c27 in term 1.
I20260926 13:40:22.571033 8252 raft_consensus.cc:2454] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e0ce36f39e9402883d8942956b16c27 in term 1.
I20260926 13:40:22.571028 8323 leader_election.cc:302] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d6d78df9f445f5852b6f10003d12da, 3e0ce36f39e9402883d8942956b16c27; no voters:
I20260926 13:40:22.571262 8616 raft_consensus.cc:2781] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:22.571507 8616 raft_consensus.cc:686] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [term 1 LEADER]: Becoming Leader. State: Replica: 3e0ce36f39e9402883d8942956b16c27, State: Running, Role: LEADER
I20260926 13:40:22.571964 8616 consensus_queue.cc:237] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } } peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } }
I20260926 13:40:22.575472 7334 catalog_manager.cc:5495] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e0ce36f39e9402883d8942956b16c27 (127.1.49.2). New cstate: current_term: 1 leader_uuid: "3e0ce36f39e9402883d8942956b16c27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } health_report { overall_health: HEALTHY } } }
W20260926 13:40:22.586225 8577 tablet.cc:2369] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:40:22.596063 8438 tablet.cc:2369] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:40:22.605099 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.600852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44164 (local address 127.1.49.2:34249)
0926 13:40:22.601009 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.601014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.601031 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:40:22.601219 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.601222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.601305 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.601442 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:40:22.601737 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.602353 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.603319 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.603564 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.603693 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.604089 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:40:22.604580 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:40:22.604744 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.604813 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.604885 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 13:40:22.612008 8528 raft_consensus.cc:1261] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da [term 1 FOLLOWER]: Refusing update from remote peer 3e0ce36f39e9402883d8942956b16c27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:22.612016 8252 raft_consensus.cc:1261] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e [term 1 FOLLOWER]: Refusing update from remote peer 3e0ce36f39e9402883d8942956b16c27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:22.612950 8617 consensus_queue.cc:1035] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:22.613162 8616 consensus_queue.cc:1035] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 13:40:22.620350 8301 tablet.cc:2369] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:40:22.624444 8621 mvcc.cc:205] Tried to move back new op lower bound from 7333601372610146304 to 7333601372457717760. Current Snapshot: MvccSnapshot[applied={T|T < 7333601372610146304}]
I20260926 13:40:22.635511 7334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:22.635682 7334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:22.641547 7334 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca6d743256e747e7bd97ca28676150d4: Sending DeleteTablet for 3 replicas of tablet e6c2c1654fec44cfa04d0c9dffb57e12
I20260926 13:40:22.642611 8508 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c2c1654fec44cfa04d0c9dffb57e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:22 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:37032
I20260926 13:40:22.642680 8367 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c2c1654fec44cfa04d0c9dffb57e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:22 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:44152
I20260926 13:40:22.643222 8631 tablet_replica.cc:330] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27: stopping tablet replica
I20260926 13:40:22.643556 8232 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c2c1654fec44cfa04d0c9dffb57e12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:22 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:57148
I20260926 13:40:22.643666 8631 raft_consensus.cc:2229] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:40:22.644064 8632 tablet_replica.cc:330] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e: stopping tablet replica
I20260926 13:40:22.644140 8630 tablet_replica.cc:330] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da: stopping tablet replica
I20260926 13:40:22.644362 8632 raft_consensus.cc:2229] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:22.644451 8630 raft_consensus.cc:2229] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:22.644706 8632 raft_consensus.cc:2258] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:22.644759 8630 raft_consensus.cc:2258] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:22.646210 8631 raft_consensus.cc:2258] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:22.646337 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 7303
I20260926 13:40:22.647475 8632 ts_tablet_manager.cc:1905] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:22.648358 8630 ts_tablet_manager.cc:1905] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:22.650705 8632 ts_tablet_manager.cc:1918] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:22.650840 8632 log.cc:1196] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-0/wal/wals/e6c2c1654fec44cfa04d0c9dffb57e12
I20260926 13:40:22.651203 8632 ts_tablet_manager.cc:1939] T e6c2c1654fec44cfa04d0c9dffb57e12 P 04dccfca386341c08195a928a2aecd0e: Deleting consensus metadata
I20260926 13:40:22.654539 8631 ts_tablet_manager.cc:1905] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 13:40:22.655017 8188 connection.cc:535] server connection from 127.0.0.1:57148 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:40:22.652969 8630 ts_tablet_manager.cc:1918] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:22.657310 8630 log.cc:1196] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/e6c2c1654fec44cfa04d0c9dffb57e12
I20260926 13:40:22.657713 8630 ts_tablet_manager.cc:1939] T e6c2c1654fec44cfa04d0c9dffb57e12 P 33d6d78df9f445f5852b6f10003d12da: Deleting consensus metadata
I20260926 13:40:22.657992 8631 ts_tablet_manager.cc:1918] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:22.658110 8631 log.cc:1196] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/e6c2c1654fec44cfa04d0c9dffb57e12
I20260926 13:40:22.658390 8631 ts_tablet_manager.cc:1939] T e6c2c1654fec44cfa04d0c9dffb57e12 P 3e0ce36f39e9402883d8942956b16c27: Deleting consensus metadata
W20260926 13:40:22.659029 8463 connection.cc:441] server connection from 127.0.0.1:37032 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37032 (request call id 1) could send its response
W20260926 13:40:22.659333 8324 connection.cc:441] server connection from 127.0.0.1:44152 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44152 (request call id 1) could send its response
I20260926 13:40:22.669607 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.645311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.1.49.61:34937)
0926 13:40:22.645408 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.645412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.651252 (+ 5840us) server_negotiation.cc:407] Connection header received
0926 13:40:22.651566 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.651572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.651643 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.651705 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:22.659271 (+ 7566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.660236 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.661634 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.662046 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.662128 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.664381 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:22.664385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:22.664388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:22.664421 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:22.666854 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.668057 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.668061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.668063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.668163 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.668510 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.668514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.668526 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.668754 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:22.668882 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.669273 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.669420 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:40:22.671029 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.659701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52988 (local address 127.1.49.60:34631)
0926 13:40:22.659858 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.659863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.660229 (+ 366us) server_negotiation.cc:407] Connection header received
0926 13:40:22.660352 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.660355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.660406 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.660504 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:22.661595 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.662105 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.662921 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.663155 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.663268 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.665324 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:22.665327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:22.665330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:22.665359 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:22.668380 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.668846 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.668849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.668852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.668945 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.669262 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.669266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.669268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.670051 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:22.670182 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.670743 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.670864 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:40:22.676211 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.664802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.1.49.61:34937)
0926 13:40:22.664952 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.664957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.665751 (+ 794us) server_negotiation.cc:407] Connection header received
0926 13:40:22.665871 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.665874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.665926 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.666003 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:22.666722 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.667241 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.668521 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.668896 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.668985 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.672216 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:22.672220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:22.672222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:22.672247 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:22.673745 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.674181 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.674185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.674187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.674271 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.674722 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.674725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.674726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.674934 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:22.675056 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.675985 (+ 929us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.676092 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 13:40:22.680507 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:22.645167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.1.49.60:34631)
0926 13:40:22.645307 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:22.645312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:22.670199 (+ 24887us) server_negotiation.cc:407] Connection header received
0926 13:40:22.670390 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:22.670394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:22.670445 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:22.670500 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:22.672087 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.672884 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.674054 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:22.674437 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:22.674519 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:22.676657 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:22.676661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:22.676664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:22.676697 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:22.678454 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.678992 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.678996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.678998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.679092 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:22.679418 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:22.679421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:22.679423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:22.679647 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:22.679769 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:22.680207 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 13:40:22.680316 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 13:40:23.626703 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:23.626149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.3:60813)
0926 13:40:23.626445 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:23.626557 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":140,"threads_started":1}
W20260926 13:40:23.627233 8575 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:36609 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
I20260926 13:40:23.627350 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:23.626764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.2:37041)
0926 13:40:23.627071 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:23.627151 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":79,"threads_started":1}
W20260926 13:40:23.627707 8436 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:36609 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
I20260926 13:40:23.630455 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:23.630066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.1:48667)
0926 13:40:23.630284 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:23.630368 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
W20260926 13:40:23.630811 8299 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:36609 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
I20260926 13:40:24.141850 8644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca6d743256e747e7bd97ca28676150d4)
I20260926 13:40:24.142107 8644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } }
I20260926 13:40:24.142454 8645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca6d743256e747e7bd97ca28676150d4)
I20260926 13:40:24.142637 8645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } }
I20260926 13:40:24.143484 8644 leader_election.cc:288] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca6d743256e747e7bd97ca28676150d4 (127.1.49.62:36609), 1428d554ea2b485b80bcbc9048734f52 (127.1.49.61:34937)
I20260926 13:40:24.143656 8645 leader_election.cc:288] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca6d743256e747e7bd97ca28676150d4 (127.1.49.62:36609), 753d614d1cb643beaa988ad729abba2e (127.1.49.60:34631)
I20260926 13:40:24.144402 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.143571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.0.0.1:58240)
0926 13:40:24.144193 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.144287 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":70,"threads_started":1}
W20260926 13:40:24.144588 8105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
I20260926 13:40:24.144901 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.143784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.0.0.1:58242)
0926 13:40:24.144727 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.144794 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":61,"threads_started":1}
W20260926 13:40:24.145147 7922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
I20260926 13:40:24.146087 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.145804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.0.0.1:58258)
0926 13:40:24.145951 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.146008 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 13:40:24.146131 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.145856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.0.0.1:58272)
0926 13:40:24.146012 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.146059 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260926 13:40:24.146215 7922 leader_election.cc:334] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ca6d743256e747e7bd97ca28676150d4 (127.1.49.62:36609): Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
W20260926 13:40:24.146229 8105 leader_election.cc:334] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ca6d743256e747e7bd97ca28676150d4 (127.1.49.62:36609): Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
I20260926 13:40:24.149062 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.143928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:34937 (local address 127.0.0.1:37986)
0926 13:40:24.144217 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.144230 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:24.144338 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:24.144748 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:24.144751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:24.144774 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:24.144923 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:24.144931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.146053 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.146056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:24.147561 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:24.147570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.148607 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.148611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:24.148690 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.148713 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:40:24.148784 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 13:40:24.148899 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":50,"threads_started":1}
I20260926 13:40:24.149080 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.144089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37986 (local address 127.1.49.61:34937)
0926 13:40:24.144339 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:24.144347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:24.144367 (+ 20us) server_negotiation.cc:407] Connection header received
0926 13:40:24.144465 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:24.144471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:24.144528 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:24.144614 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:24.145058 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.145926 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.147716 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.148457 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.148584 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.148657 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:24.148835 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 13:40:24.148942 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260926 13:40:24.150249 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.143884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:34631 (local address 127.0.0.1:55976)
0926 13:40:24.144510 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.144525 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:24.144604 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:24.145497 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:24.145502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:24.145524 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:24.145613 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:24.145619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.147391 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.147394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:24.148782 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:24.148797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.149765 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.149769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:24.149915 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.149944 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 13:40:24.150006 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:40:24.150117 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":67,"threads_started":1}
I20260926 13:40:24.150216 7944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "753d614d1cb643beaa988ad729abba2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1428d554ea2b485b80bcbc9048734f52" is_pre_election: true
I20260926 13:40:24.150456 7944 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 753d614d1cb643beaa988ad729abba2e in term 1.
I20260926 13:40:24.150892 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.144043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55976 (local address 127.1.49.60:34631)
0926 13:40:24.144658 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:24.144663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:24.144722 (+ 59us) server_negotiation.cc:407] Connection header received
0926 13:40:24.144759 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:24.144762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:24.144797 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:24.145755 (+ 958us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:24.145976 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.147154 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.148963 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.149595 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.150591 (+ 996us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.150661 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:24.150703 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 13:40:24.150783 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":69,"threads_started":1}
I20260926 13:40:24.150851 8105 leader_election.cc:302] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1428d554ea2b485b80bcbc9048734f52, 753d614d1cb643beaa988ad729abba2e; no voters: ca6d743256e747e7bd97ca28676150d4
I20260926 13:40:24.151327 8644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 13:40:24.151405 8644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca6d743256e747e7bd97ca28676150d4)
I20260926 13:40:24.151379 8129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1428d554ea2b485b80bcbc9048734f52" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "753d614d1cb643beaa988ad729abba2e" is_pre_election: true
I20260926 13:40:24.151553 8644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:40:24.152740 8644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } }
I20260926 13:40:24.153064 8129 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1428d554ea2b485b80bcbc9048734f52 in current term 2: Already voted for candidate 753d614d1cb643beaa988ad729abba2e in this term.
I20260926 13:40:24.153271 8644 leader_election.cc:288] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [CANDIDATE]: Term 2 election: Requested vote from peers ca6d743256e747e7bd97ca28676150d4 (127.1.49.62:36609), 1428d554ea2b485b80bcbc9048734f52 (127.1.49.61:34937)
I20260926 13:40:24.153718 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.153405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.0.0.1:58302)
0926 13:40:24.153547 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.153641 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 13:40:24.153864 7944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "753d614d1cb643beaa988ad729abba2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1428d554ea2b485b80bcbc9048734f52"
I20260926 13:40:24.153976 7920 leader_election.cc:302] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1428d554ea2b485b80bcbc9048734f52; no voters: 753d614d1cb643beaa988ad729abba2e, ca6d743256e747e7bd97ca28676150d4
I20260926 13:40:24.153995 7944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:40:24.154860 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.154525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.0.0.1:58312)
0926 13:40:24.154667 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.154769 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 13:40:24.155031 8105 leader_election.cc:334] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ca6d743256e747e7bd97ca28676150d4 (127.1.49.62:36609): Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
I20260926 13:40:24.155408 7944 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 753d614d1cb643beaa988ad729abba2e in term 2.
I20260926 13:40:24.155566 8645 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:40:24.155867 8105 leader_election.cc:302] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1428d554ea2b485b80bcbc9048734f52, 753d614d1cb643beaa988ad729abba2e; no voters: ca6d743256e747e7bd97ca28676150d4
I20260926 13:40:24.156057 8644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:40:24.156314 8644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [term 2 LEADER]: Becoming Leader. State: Replica: 753d614d1cb643beaa988ad729abba2e, State: Running, Role: LEADER
I20260926 13:40:24.156643 8644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } }
I20260926 13:40:24.158147 8654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 753d614d1cb643beaa988ad729abba2e. Latest consensus state: current_term: 2 leader_uuid: "753d614d1cb643beaa988ad729abba2e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } } }
I20260926 13:40:24.158305 8654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:24.158663 8656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:24.161031 8656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:24.161449 8656 catalog_manager.cc:1259] Loaded cluster ID: f166fbec81484299be1710d5dd29f21d
I20260926 13:40:24.161502 8656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:24.162072 8656 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:24.162474 8656 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: Loaded TSK: 0
I20260926 13:40:24.163183 8656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:40:24.212347 8119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:40:24.216187 8119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:40:24.219887 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.219512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.0.0.1:58332)
0926 13:40:24.219690 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.219743 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260926 13:40:24.219897 7944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [term 2 FOLLOWER]: Refusing update from remote peer 753d614d1cb643beaa988ad729abba2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 13:40:24.220731 8654 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 13:40:24.221035 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.220782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.0.0.1:58338)
0926 13:40:24.220912 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.220968 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 13:40:24.221140 8105 consensus_peers.cc:487] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e -> Peer ca6d743256e747e7bd97ca28676150d4 (127.1.49.62:36609): Couldn't send request to peer ca6d743256e747e7bd97ca28676150d4. Status: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 13:40:24.222479 8645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 753d614d1cb643beaa988ad729abba2e. Latest consensus state: current_term: 2 leader_uuid: "753d614d1cb643beaa988ad729abba2e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } } }
I20260926 13:40:24.222618 8645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:40:24.223937 8644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "753d614d1cb643beaa988ad729abba2e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } } }
I20260926 13:40:24.224089 8644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:24.224277 8658 mvcc.cc:205] Tried to move back new op lower bound from 7333601379198103552 to 7333601378948296704. Current Snapshot: MvccSnapshot[applied={T|T < 7333601379198103552}]
I20260926 13:40:24.224537 8645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "753d614d1cb643beaa988ad729abba2e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca6d743256e747e7bd97ca28676150d4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36609 } } peers { permanent_uuid: "1428d554ea2b485b80bcbc9048734f52" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 34937 } attrs { promote: false } } peers { permanent_uuid: "753d614d1cb643beaa988ad729abba2e" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 34631 } attrs { promote: false } } }
I20260926 13:40:24.224647 8645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1428d554ea2b485b80bcbc9048734f52 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:40:24.235512 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.230779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.1.49.2:34249)
0926 13:40:24.231929 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:24.231933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:24.231946 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:40:24.232003 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:24.232006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:24.232051 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:24.232139 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:24.232383 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.233170 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.234326 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.234917 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.235049 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.235123 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:24.235283 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 13:40:24.235376 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1065,"thread_start_us":80,"threads_started":1}
I20260926 13:40:24.235512 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.230672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:34249 (local address 127.0.0.1:60462)
0926 13:40:24.231074 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.231090 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:24.231164 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:24.232146 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:24.232148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:24.232195 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:24.232265 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:24.232270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.233259 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.233265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:24.234182 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:24.234192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.235044 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.235048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:24.235143 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.235166 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:40:24.235240 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 13:40:24.235326 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":35}
I20260926 13:40:24.235526 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.230914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:42317 (local address 127.0.0.1:37156)
0926 13:40:24.231265 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.231279 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:24.231374 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:24.231854 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:24.231857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:24.231875 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:24.231997 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:24.232004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.232896 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.232902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:24.234161 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:24.234171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.235115 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.235119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:24.235208 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.235248 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 13:40:24.235296 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 13:40:24.235367 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":46}
I20260926 13:40:24.235831 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.231141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:45087 (local address 127.0.0.1:53238)
0926 13:40:24.231458 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.231472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:24.231554 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:24.232008 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:24.232011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:24.232027 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:24.232137 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:24.232144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.233207 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.233210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:24.234394 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:24.234401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.235428 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.235433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:24.235513 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.235534 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:40:24.235601 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:40:24.235683 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":32,"thread_start_us":111,"threads_started":1}
I20260926 13:40:24.236236 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.231508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53238 (local address 127.1.49.1:45087)
0926 13:40:24.231736 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:24.231739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:24.231751 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:40:24.231857 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:24.231860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:24.231906 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:24.231996 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:24.232272 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.233090 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.234521 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.235192 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.235305 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.235373 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:24.235987 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 13:40:24.236099 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":65,"threads_started":1}
I20260926 13:40:24.236243 8367 tablet_service.cc:1467] Processing CreateTablet for tablet 82e4393f880f43f79653e49f05a89947 (DEFAULT_TABLE table=test-table [id=64c48e7887a1454eba974434ce28196b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:24.236603 8367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e4393f880f43f79653e49f05a89947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:24.236830 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.231041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.1.49.3:42317)
0926 13:40:24.231585 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:24.231591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:24.231607 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:40:24.231657 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:24.231660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:24.231712 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:24.231822 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:24.232129 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.232779 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.234297 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.234977 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.236179 (+ 1202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.236247 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:24.236583 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 13:40:24.236693 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":96,"threads_started":1}
I20260926 13:40:24.236799 8232 tablet_service.cc:1467] Processing CreateTablet for tablet 82e4393f880f43f79653e49f05a89947 (DEFAULT_TABLE table=test-table [id=64c48e7887a1454eba974434ce28196b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:24.237071 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e4393f880f43f79653e49f05a89947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:24.239420 8666 tablet_bootstrap.cc:492] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27: Bootstrap starting.
I20260926 13:40:24.239835 8508 tablet_service.cc:1467] Processing CreateTablet for tablet 82e4393f880f43f79653e49f05a89947 (DEFAULT_TABLE table=test-table [id=64c48e7887a1454eba974434ce28196b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:24.240113 8508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e4393f880f43f79653e49f05a89947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:24.240460 8666 tablet_bootstrap.cc:654] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:24.241525 8667 tablet_bootstrap.cc:492] T 82e4393f880f43f79653e49f05a89947 P 04dccfca386341c08195a928a2aecd0e: Bootstrap starting.
I20260926 13:40:24.241833 8666 tablet_bootstrap.cc:492] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27: No bootstrap required, opened a new log
I20260926 13:40:24.241922 8666 ts_tablet_manager.cc:1397] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:40:24.242430 8667 tablet_bootstrap.cc:654] T 82e4393f880f43f79653e49f05a89947 P 04dccfca386341c08195a928a2aecd0e: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:24.242393 8666 raft_consensus.cc:348] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } } peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } }
I20260926 13:40:24.242522 8666 raft_consensus.cc:374] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:24.242558 8666 raft_consensus.cc:729] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0ce36f39e9402883d8942956b16c27, State: Initialized, Role: FOLLOWER
I20260926 13:40:24.242678 8669 tablet_bootstrap.cc:492] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da: Bootstrap starting.
I20260926 13:40:24.242702 8666 consensus_queue.cc:260] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } } peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } }
I20260926 13:40:24.243072 8666 ts_tablet_manager.cc:1428] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:24.243675 8667 tablet_bootstrap.cc:492] T 82e4393f880f43f79653e49f05a89947 P 04dccfca386341c08195a928a2aecd0e: No bootstrap required, opened a new log
I20260926 13:40:24.243760 8667 ts_tablet_manager.cc:1397] T 82e4393f880f43f79653e49f05a89947 P 04dccfca386341c08195a928a2aecd0e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:40:24.243834 8669 tablet_bootstrap.cc:654] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:24.244274 8667 raft_consensus.cc:348] T 82e4393f880f43f79653e49f05a89947 P 04dccfca386341c08195a928a2aecd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } } peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } }
I20260926 13:40:24.244391 8667 raft_consensus.cc:374] T 82e4393f880f43f79653e49f05a89947 P 04dccfca386341c08195a928a2aecd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:24.244431 8667 raft_consensus.cc:729] T 82e4393f880f43f79653e49f05a89947 P 04dccfca386341c08195a928a2aecd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04dccfca386341c08195a928a2aecd0e, State: Initialized, Role: FOLLOWER
I20260926 13:40:24.244565 8667 consensus_queue.cc:260] T 82e4393f880f43f79653e49f05a89947 P 04dccfca386341c08195a928a2aecd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } } peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } }
I20260926 13:40:24.244913 8667 ts_tablet_manager.cc:1428] T 82e4393f880f43f79653e49f05a89947 P 04dccfca386341c08195a928a2aecd0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:24.244971 8669 tablet_bootstrap.cc:492] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da: No bootstrap required, opened a new log
I20260926 13:40:24.245046 8669 ts_tablet_manager.cc:1397] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:40:24.245455 8669 raft_consensus.cc:348] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } } peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } }
I20260926 13:40:24.245556 8435 heartbeater.cc:503] Master 127.1.49.60:34631 was elected leader, sending a full tablet report...
I20260926 13:40:24.245579 8669 raft_consensus.cc:374] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:24.245611 8669 raft_consensus.cc:729] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d6d78df9f445f5852b6f10003d12da, State: Initialized, Role: FOLLOWER
I20260926 13:40:24.245754 8669 consensus_queue.cc:260] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } } peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } }
I20260926 13:40:24.246130 8669 ts_tablet_manager.cc:1428] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:24.246642 8672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.243748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.2:56243)
0926 13:40:24.246466 (+ 2718us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.246534 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2643,"thread_start_us":123,"threads_started":1}
I20260926 13:40:24.247428 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.247062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.3:46605)
0926 13:40:24.247276 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.247335 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260926 13:40:24.247471 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.245644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.1:59795)
0926 13:40:24.247278 (+ 1634us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.247342 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1561,"thread_start_us":97,"threads_started":1}
I20260926 13:40:24.248502 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.248012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.1:56181)
0926 13:40:24.248375 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.248430 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 13:40:24.249305 8298 heartbeater.cc:503] Master 127.1.49.60:34631 was elected leader, sending a full tablet report...
I20260926 13:40:24.249873 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.248288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.3:42635)
0926 13:40:24.249721 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.249806 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260926 13:40:24.249884 8672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.249152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.2:46447)
0926 13:40:24.249743 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.249814 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517}
I20260926 13:40:24.249897 8574 heartbeater.cc:503] Master 127.1.49.60:34631 was elected leader, sending a full tablet report...
I20260926 13:40:24.414166 8676 raft_consensus.cc:482] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:40:24.414322 8676 raft_consensus.cc:504] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } } peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } }
I20260926 13:40:24.415462 8676 leader_election.cc:288] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e0ce36f39e9402883d8942956b16c27 (127.1.49.2:34249), 04dccfca386341c08195a928a2aecd0e (127.1.49.1:45087)
I20260926 13:40:24.419266 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.415733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:45087 (local address 127.1.49.3:59449)
0926 13:40:24.415951 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.415980 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:24.416114 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:24.416427 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:24.416430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:24.416447 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:24.416557 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:24.416562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.417310 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.417314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:24.418125 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:24.418134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.418822 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.418827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:24.418934 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.418962 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 13:40:24.419050 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 13:40:24.419123 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260926 13:40:24.419266 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.415935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:59449 (local address 127.1.49.1:45087)
0926 13:40:24.416067 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:24.416072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:24.416087 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:40:24.416239 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:24.416244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:24.416298 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:24.416401 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:24.416665 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.417185 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.418264 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.418685 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.418815 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.418896 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:24.419066 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 13:40:24.419139 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:40:24.419485 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.415749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:34249 (local address 127.1.49.3:46273)
0926 13:40:24.416208 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.416220 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:24.416436 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:24.416734 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:24.416737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:24.416752 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:24.416846 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:24.416857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.417584 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.417589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:24.418351 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:24.418360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.419020 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.419024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:24.419119 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.419147 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 13:40:24.419234 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 13:40:24.419304 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":172,"threads_started":1}
I20260926 13:40:24.419884 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.415914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:46273 (local address 127.1.49.2:34249)
0926 13:40:24.416055 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:24.416060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:24.416312 (+ 252us) server_negotiation.cc:407] Connection header received
0926 13:40:24.416573 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:24.416581 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:24.416628 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:24.416722 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:24.416966 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.417461 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.418479 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.418892 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.419016 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.419075 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:24.419250 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 13:40:24.419323 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:40:24.419827 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e4393f880f43f79653e49f05a89947" candidate_uuid: "33d6d78df9f445f5852b6f10003d12da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04dccfca386341c08195a928a2aecd0e" is_pre_election: true
I20260926 13:40:24.419993 8252 raft_consensus.cc:2454] T 82e4393f880f43f79653e49f05a89947 P 04dccfca386341c08195a928a2aecd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d6d78df9f445f5852b6f10003d12da in term 0.
I20260926 13:40:24.420354 8389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e4393f880f43f79653e49f05a89947" candidate_uuid: "33d6d78df9f445f5852b6f10003d12da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0ce36f39e9402883d8942956b16c27" is_pre_election: true
I20260926 13:40:24.420424 8460 leader_election.cc:302] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04dccfca386341c08195a928a2aecd0e, 33d6d78df9f445f5852b6f10003d12da; no voters:
I20260926 13:40:24.420601 8389 raft_consensus.cc:2454] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d6d78df9f445f5852b6f10003d12da in term 0.
I20260926 13:40:24.420713 8676 raft_consensus.cc:2781] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:40:24.420760 8676 raft_consensus.cc:482] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:40:24.420807 8676 raft_consensus.cc:3037] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:24.421893 8676 raft_consensus.cc:504] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } } peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } }
I20260926 13:40:24.422292 8676 leader_election.cc:288] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [CANDIDATE]: Term 1 election: Requested vote from peers 3e0ce36f39e9402883d8942956b16c27 (127.1.49.2:34249), 04dccfca386341c08195a928a2aecd0e (127.1.49.1:45087)
I20260926 13:40:24.422662 8389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e4393f880f43f79653e49f05a89947" candidate_uuid: "33d6d78df9f445f5852b6f10003d12da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0ce36f39e9402883d8942956b16c27"
I20260926 13:40:24.422780 8389 raft_consensus.cc:3037] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:24.422741 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e4393f880f43f79653e49f05a89947" candidate_uuid: "33d6d78df9f445f5852b6f10003d12da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04dccfca386341c08195a928a2aecd0e"
I20260926 13:40:24.422894 8252 raft_consensus.cc:3037] T 82e4393f880f43f79653e49f05a89947 P 04dccfca386341c08195a928a2aecd0e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:24.423792 8389 raft_consensus.cc:2454] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d6d78df9f445f5852b6f10003d12da in term 1.
I20260926 13:40:24.423904 8252 raft_consensus.cc:2454] T 82e4393f880f43f79653e49f05a89947 P 04dccfca386341c08195a928a2aecd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d6d78df9f445f5852b6f10003d12da in term 1.
I20260926 13:40:24.424145 8461 leader_election.cc:302] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d6d78df9f445f5852b6f10003d12da, 3e0ce36f39e9402883d8942956b16c27; no voters:
I20260926 13:40:24.424309 8676 raft_consensus.cc:2781] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:24.424530 8676 raft_consensus.cc:686] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [term 1 LEADER]: Becoming Leader. State: Replica: 33d6d78df9f445f5852b6f10003d12da, State: Running, Role: LEADER
I20260926 13:40:24.424784 8676 consensus_queue.cc:237] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } } peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } }
I20260926 13:40:24.425359 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.425091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.3:48097)
0926 13:40:24.425222 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.425293 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 13:40:24.427610 8118 catalog_manager.cc:5495] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da reported cstate change: term changed from 0 to 1, leader changed from <none> to 33d6d78df9f445f5852b6f10003d12da (127.1.49.3). New cstate: current_term: 1 leader_uuid: "33d6d78df9f445f5852b6f10003d12da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d6d78df9f445f5852b6f10003d12da" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 } health_report { overall_health: UNKNOWN } } }
I20260926 13:40:24.477509 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.473220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.1.49.3:42317)
0926 13:40:24.473372 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:24.473376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:24.473391 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:40:24.473550 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:24.473553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:24.473611 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:24.473705 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:40:24.473990 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.474512 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.475725 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:24.475995 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:24.476092 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:24.476475 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:40:24.477033 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:40:24.477139 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:24.477234 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 13:40:24.477305 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 13:40:24.481169 8252 raft_consensus.cc:1261] T 82e4393f880f43f79653e49f05a89947 P 04dccfca386341c08195a928a2aecd0e [term 1 FOLLOWER]: Refusing update from remote peer 33d6d78df9f445f5852b6f10003d12da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:24.481518 8389 raft_consensus.cc:1261] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27 [term 1 FOLLOWER]: Refusing update from remote peer 33d6d78df9f445f5852b6f10003d12da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:24.481896 8676 consensus_queue.cc:1035] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [LEADER]: Connected to new peer: Peer: permanent_uuid: "04dccfca386341c08195a928a2aecd0e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:24.482173 8679 consensus_queue.cc:1035] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e0ce36f39e9402883d8942956b16c27" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:24.484443 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.484053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.1:48841)
0926 13:40:24.484267 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.484350 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260926 13:40:24.484459 8672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.484092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.2:43695)
0926 13:40:24.484349 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.484396 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260926 13:40:24.488046 8685 mvcc.cc:205] Tried to move back new op lower bound from 7333601380268023808 to 7333601380046520320. Current Snapshot: MvccSnapshot[applied={T|T < 7333601380268023808}]
I20260926 13:40:24.488139 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.487872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.3:50057)
0926 13:40:24.487979 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.488045 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 13:40:24.489442 8672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.489176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.2:55221)
0926 13:40:24.489277 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.489350 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 13:40:24.492089 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.491565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.3:56573)
0926 13:40:24.491923 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.491994 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266}
I20260926 13:40:24.493532 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.493105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36609 (local address 127.1.49.1:53373)
0926 13:40:24.493332 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.493414 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260926 13:40:24.499306 8118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:24.499502 8118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:24.504321 8118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 753d614d1cb643beaa988ad729abba2e: Sending DeleteTablet for 3 replicas of tablet 82e4393f880f43f79653e49f05a89947
I20260926 13:40:24.505211 8232 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e4393f880f43f79653e49f05a89947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:24 UTC) from {username='oryx', principal='oryx/127.1.49.60@KRBTEST.COM'} at 127.0.0.1:53238
I20260926 13:40:24.505383 8508 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e4393f880f43f79653e49f05a89947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:24 UTC) from {username='oryx', principal='oryx/127.1.49.60@KRBTEST.COM'} at 127.0.0.1:37156
I20260926 13:40:24.505959 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 8080
I20260926 13:40:24.506178 8367 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e4393f880f43f79653e49f05a89947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:24 UTC) from {username='oryx', principal='oryx/127.1.49.60@KRBTEST.COM'} at 127.0.0.1:60462
I20260926 13:40:24.506595 8695 tablet_replica.cc:330] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da: stopping tablet replica
I20260926 13:40:24.506896 8695 raft_consensus.cc:2229] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:40:24.507296 8695 raft_consensus.cc:2258] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:24.508093 8695 ts_tablet_manager.cc:1905] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:24.510824 8695 ts_tablet_manager.cc:1918] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:24.510913 8695 log.cc:1196] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/82e4393f880f43f79653e49f05a89947
I20260926 13:40:24.511205 8695 ts_tablet_manager.cc:1939] T 82e4393f880f43f79653e49f05a89947 P 33d6d78df9f445f5852b6f10003d12da: Deleting consensus metadata
I20260926 13:40:24.512254 8104 catalog_manager.cc:4841] TS 33d6d78df9f445f5852b6f10003d12da (127.1.49.3:42317): tablet 82e4393f880f43f79653e49f05a89947 (table test-table [id=64c48e7887a1454eba974434ce28196b]) successfully deleted
I20260926 13:40:24.514168 8696 tablet_replica.cc:330] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27: stopping tablet replica
I20260926 13:40:24.514324 8696 raft_consensus.cc:2229] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:24.514505 8696 raft_consensus.cc:2258] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 13:40:24.515060 8664 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:24.514256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:45087 (local address 127.0.0.1:53240)
0926 13:40:24.514453 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.514470 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:24.514567 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:24.514944 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:45087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":107}
W20260926 13:40:24.515277 8103 catalog_manager.cc:4584] TS 04dccfca386341c08195a928a2aecd0e (127.1.49.1:45087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82e4393f880f43f79653e49f05a89947: Network error: Client connection negotiation failed: client connection to 127.1.49.1:45087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:40:24.515386 8696 ts_tablet_manager.cc:1905] T 82e4393f880f43f79653e49f05a89947 P 3e0ce36f39e9402883d8942956b16c27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:24.515889 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 8306
I20260926 13:40:24.523671 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 8442
I20260926 13:40:24.524928 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:24.524038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:34249 (local address 127.0.0.1:60476)
0926 13:40:24.524732 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:24.524803 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.2:34249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586}
I20260926 13:40:24.530007 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 7799
I20260926 13:40:24.536080 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 7907
2026-09-26T13:40:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5876 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:40:24 dist-test-slave-1wns krb5kdc[8699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:40:24 dist-test-slave-1wns krb5kdc[8699](info): set up 2 sockets
Sep 26 13:40:24 dist-test-slave-1wns krb5kdc[8699](info): commencing operation
krb5kdc: starting...
W20260926 13:40:25.649662 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:40:25 dist-test-slave-1wns krb5kdc[8699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:40:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.62.keytab.
Entry for principal oryx/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.62.keytab.
I20260926 13:40:25.791940 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:41433
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.62.keytab
--principal=oryx/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:41433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:25.912050 8712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:25.912370 8712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:25.912423 8712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:25.916916 8712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:25.917001 8712 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 13:40:25.917019 8712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:25.917038 8712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:25.917055 8712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:25.917101 8712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:25.923214 8712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:41433
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.62.keytab
--principal=oryx/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:41433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:25.924806 8712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:25.926101 8712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:40:25.936287 8712 server_base.cc:1034] running on GCE node
W20260926 13:40:25.936211 8718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:25.936295 8717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:25.936228 8720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:25.937256 8712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:25.938601 8712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:25.939783 8712 hybrid_clock.cc:648] HybridClock initialized: now 1790430025939775 us; error 37 us; skew 500 ppm
Sep 26 13:40:25 dist-test-slave-1wns krb5kdc[8699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430025, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:25.950384 8712 init.cc:377] Logged in from keytab as oryx/127.1.49.62@KRBTEST.COM (short username oryx)
I20260926 13:40:25.951550 8712 webserver.cc:460] Webserver started at http://127.1.49.62:42523/ using document root <none> and password file <none>
I20260926 13:40:25.952298 8712 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:25.952450 8712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:25.952720 8712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:25.955359 8712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "c624eaa121ab4590a69af5dc5e9d5c9b"
format_stamp: "Formatted at 2026-09-26 13:40:25 on dist-test-slave-1wns"
server_key: "01a7e9786d549c1069be2e6a498c318c"
server_key_iv: "792636a42007d4a109c383662ac5404a"
server_key_version: "encryptionkey@0"
I20260926 13:40:25.956102 8712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "c624eaa121ab4590a69af5dc5e9d5c9b"
format_stamp: "Formatted at 2026-09-26 13:40:25 on dist-test-slave-1wns"
server_key: "01a7e9786d549c1069be2e6a498c318c"
server_key_iv: "792636a42007d4a109c383662ac5404a"
server_key_version: "encryptionkey@0"
I20260926 13:40:25.960987 8712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 13:40:25.963968 8727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:25.965559 8712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 13:40:25.965739 8712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "c624eaa121ab4590a69af5dc5e9d5c9b"
format_stamp: "Formatted at 2026-09-26 13:40:25 on dist-test-slave-1wns"
server_key: "01a7e9786d549c1069be2e6a498c318c"
server_key_iv: "792636a42007d4a109c383662ac5404a"
server_key_version: "encryptionkey@0"
I20260926 13:40:25.965883 8712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:25.976922 8712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:25.980471 8712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:25.980687 8712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:25.989029 8712 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:41433
I20260926 13:40:25.989045 8789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:41433 every 8 connection(s)
I20260926 13:40:25.990124 8712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:25.993232 8790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:25.999231 8790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b: Bootstrap starting.
I20260926 13:40:25.999655 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 8712
I20260926 13:40:25.999788 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:26.000135 1220 external_mini_cluster.cc:1428] Setting key 2b8dc352477eb63a4394044063a61ba6
I20260926 13:40:26.002342 8790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:26.003165 8790 log.cc:824] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:26.005699 8790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b: No bootstrap required, opened a new log
Sep 26 13:40:26 dist-test-slave-1wns krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
I20260926 13:40:26.012377 8790 raft_consensus.cc:348] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c624eaa121ab4590a69af5dc5e9d5c9b" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41433 } }
I20260926 13:40:26.012778 8790 raft_consensus.cc:374] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:26.012876 8790 raft_consensus.cc:729] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c624eaa121ab4590a69af5dc5e9d5c9b, State: Initialized, Role: FOLLOWER
I20260926 13:40:26.013514 8790 consensus_queue.cc:260] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c624eaa121ab4590a69af5dc5e9d5c9b" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41433 } }
I20260926 13:40:26.013710 8790 raft_consensus.cc:388] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:26.013794 8790 raft_consensus.cc:482] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:26.013921 8790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:26.015103 8790 raft_consensus.cc:504] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c624eaa121ab4590a69af5dc5e9d5c9b" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41433 } }
I20260926 13:40:26.015591 8790 leader_election.cc:302] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c624eaa121ab4590a69af5dc5e9d5c9b; no voters:
I20260926 13:40:26.016067 8790 leader_election.cc:288] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:26.016155 8795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:26.016395 8795 raft_consensus.cc:686] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [term 1 LEADER]: Becoming Leader. State: Replica: c624eaa121ab4590a69af5dc5e9d5c9b, State: Running, Role: LEADER
I20260926 13:40:26.016775 8795 consensus_queue.cc:237] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c624eaa121ab4590a69af5dc5e9d5c9b" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41433 } }
I20260926 13:40:26.017666 8790 sys_catalog.cc:564] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:26.018810 8796 sys_catalog.cc:455] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c624eaa121ab4590a69af5dc5e9d5c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c624eaa121ab4590a69af5dc5e9d5c9b" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41433 } } }
I20260926 13:40:26.019025 8796 sys_catalog.cc:458] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:26.019428 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.001691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.1.49.62:41433)
0926 13:40:26.002192 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:26.002208 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:26.002247 (+ 39us) server_negotiation.cc:407] Connection header received
0926 13:40:26.003072 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:26.003093 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:26.003519 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:26.003986 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:26.004977 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.006114 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.007527 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.008106 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.008232 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.012487 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:26.012494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:26.012511 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:26.012551 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:26.014863 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.015506 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.015513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.015527 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.015673 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.016121 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.016126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.016128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.016858 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:26.017760 (+ 902us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:26.017935 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 13:40:26.018302 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":167,"threads_started":1}
I20260926 13:40:26.019363 8797 sys_catalog.cc:455] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c624eaa121ab4590a69af5dc5e9d5c9b. Latest consensus state: current_term: 1 leader_uuid: "c624eaa121ab4590a69af5dc5e9d5c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c624eaa121ab4590a69af5dc5e9d5c9b" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41433 } } }
I20260926 13:40:26.019594 8797 sys_catalog.cc:458] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:26.020044 8804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:26.023247 8804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:26.030166 8804 catalog_manager.cc:1347] Generated new cluster ID: 0e6131dfe65642cbad67904d2d51115c
I20260926 13:40:26.030438 8804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:26.037209 8804 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:26.038367 8804 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:26.046133 8804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b: Generated new TSK 0
I20260926 13:40:26.047150 8804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.1.keytab.
Entry for principal oryx/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.1.keytab.
I20260926 13:40:26.084671 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41433
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.1.keytab
--principal=oryx/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:26.213534 8816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:26.213848 8816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:26.213901 8816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:26.218930 8816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:26.219012 8816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:26.219058 8816 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:40:26.219096 8816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:40:26.225804 8816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.1.keytab
--principal=oryx/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.49.62:41433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:26.227020 8816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:26.228112 8816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:26.239108 8822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:26.239105 8824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:26.239105 8821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:26.239523 8816 server_base.cc:1034] running on GCE node
I20260926 13:40:26.240221 8816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:26.241036 8816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:26.242285 8816 hybrid_clock.cc:648] HybridClock initialized: now 1790430026242265 us; error 33 us; skew 500 ppm
Sep 26 13:40:26 dist-test-slave-1wns krb5kdc[8699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430026, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:26.245639 8816 init.cc:377] Logged in from keytab as oryx/127.1.49.1@KRBTEST.COM (short username oryx)
I20260926 13:40:26.246698 8816 webserver.cc:460] Webserver started at http://127.1.49.1:39775/ using document root <none> and password file <none>
I20260926 13:40:26.247346 8816 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:26.247447 8816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:26.247694 8816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:26.250056 8816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "50bcd547e39d4ded8639619ceab74179"
format_stamp: "Formatted at 2026-09-26 13:40:26 on dist-test-slave-1wns"
server_key: "d5f4127d5a0c22d5cca3ca7e2b4cf1c6"
server_key_iv: "5dd89d8cbe3eee5ba403786633c4ad44"
server_key_version: "encryptionkey@0"
I20260926 13:40:26.250941 8816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "50bcd547e39d4ded8639619ceab74179"
format_stamp: "Formatted at 2026-09-26 13:40:26 on dist-test-slave-1wns"
server_key: "d5f4127d5a0c22d5cca3ca7e2b4cf1c6"
server_key_iv: "5dd89d8cbe3eee5ba403786633c4ad44"
server_key_version: "encryptionkey@0"
I20260926 13:40:26.255414 8816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 13:40:26.258837 8831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:26.260669 8816 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:40:26.260836 8816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "50bcd547e39d4ded8639619ceab74179"
format_stamp: "Formatted at 2026-09-26 13:40:26 on dist-test-slave-1wns"
server_key: "d5f4127d5a0c22d5cca3ca7e2b4cf1c6"
server_key_iv: "5dd89d8cbe3eee5ba403786633c4ad44"
server_key_version: "encryptionkey@0"
I20260926 13:40:26.260989 8816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:26.282145 8816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:26.285748 8816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:26.286007 8816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:26.287954 8816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:26.288044 8816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:26.288134 8816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:26.288175 8816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:40:26 dist-test-slave-1wns krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430026, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
I20260926 13:40:26.300545 8816 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:42283
I20260926 13:40:26.300571 8946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:42283 every 8 connection(s)
I20260926 13:40:26.301957 8816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:40:26.302896 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 8816
I20260926 13:40:26.303045 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:40:26.303356 1220 external_mini_cluster.cc:1428] Setting key ffde3857702608ffe689e0540166dbec
I20260926 13:40:26.304255 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.288748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:57445 (local address 127.1.49.62:41433)
0926 13:40:26.288956 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:26.288961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:26.289990 (+ 1029us) server_negotiation.cc:407] Connection header received
0926 13:40:26.291504 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:26.291512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:26.291592 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:26.291662 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:26.293930 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.294557 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.295835 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.296088 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.296210 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.298874 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:26.298877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:26.298879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:26.298905 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:26.300639 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.301282 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.301286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.301287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.301441 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.301846 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.301849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.301851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.302109 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:26.302188 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:26.303963 (+ 1775us) server_negotiation.cc:299] Negotiation successful
0926 13:40:26.304108 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 13:40:26.304294 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.289233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41433 (local address 127.1.49.1:57445)
0926 13:40:26.289780 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:26.289848 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:26.291163 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:26.292223 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:26.292239 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:26.292861 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:26.293729 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.293756 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.294696 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.294699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.295684 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.295693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.296263 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.296267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.296450 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.297119 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:26.297140 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:26.298673 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:26.300771 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:26.300778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:26.300795 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:26.301157 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:26.301555 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:26.301558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:26.301561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:26.301740 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:26.302198 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:26.302205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:26.302584 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 13:40:26.302829 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 13:40:26.303226 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":120,"threads_started":1}
I20260926 13:40:26.305622 8947 heartbeater.cc:347] Connected to a master server at 127.1.49.62:41433
I20260926 13:40:26.305816 8947 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:26.306386 8947 heartbeater.cc:511] Master 127.1.49.62:41433 requested a full tablet report, sending...
I20260926 13:40:26.308331 8739 ts_manager.cc:194] Registered new tserver with Master: 50bcd547e39d4ded8639619ceab74179 (127.1.49.1:42283)
I20260926 13:40:26.309615 8739 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:57445
WARNING: no policy specified for oryx/127.1.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.49.2@KRBTEST.COM" created.
I20260926 13:40:26.324031 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.308175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:50671 (local address 127.1.49.62:41433)
0926 13:40:26.308310 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:26.308315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:26.314414 (+ 6099us) server_negotiation.cc:407] Connection header received
0926 13:40:26.314638 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:26.314641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:26.314691 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:26.314800 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:26.315378 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.316164 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.317158 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.317568 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.317698 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.319532 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:26.319536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:26.319538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:26.319566 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:26.321902 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.322510 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.322513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.322515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.322602 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.322989 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.322995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.322998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.323236 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:26.323371 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:26.323672 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 13:40:26.323835 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:40:26.324059 8954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.308107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41433 (local address 127.1.49.1:50671)
0926 13:40:26.314178 (+ 6071us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:26.314194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:26.314482 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:26.314819 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:26.314823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:26.315102 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:26.315236 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.315244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.316322 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.316326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.317033 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.317044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.317721 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.317724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.317813 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.318600 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:26.318612 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:26.319228 (+ 616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:26.322066 (+ 2838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:26.322070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:26.322072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:26.322315 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:26.322713 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:26.322717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:26.322719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:26.322832 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:26.323359 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:26.323363 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:26.323483 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 13:40:26.323608 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 13:40:26.323799 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5957,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.2.keytab.
Entry for principal oryx/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.2.keytab.
I20260926 13:40:26.336890 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41433
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.2.keytab
--principal=oryx/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:26.458364 8956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:26.458710 8956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:26.458760 8956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:26.463232 8956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:26.463339 8956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:26.463423 8956 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:40:26.463507 8956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:40:26.469429 8956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.2.keytab
--principal=oryx/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.49.62:41433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:26.470698 8956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:26.471849 8956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:40:26.482766 8956 server_base.cc:1034] running on GCE node
W20260926 13:40:26.482708 8962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:26.482708 8964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:26.482781 8961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:26.483700 8956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:26.484560 8956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:26.485782 8956 hybrid_clock.cc:648] HybridClock initialized: now 1790430026485756 us; error 48 us; skew 500 ppm
Sep 26 13:40:26 dist-test-slave-1wns krb5kdc[8699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430026, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:26.488858 8956 init.cc:377] Logged in from keytab as oryx/127.1.49.2@KRBTEST.COM (short username oryx)
I20260926 13:40:26.489816 8956 webserver.cc:460] Webserver started at http://127.1.49.2:46719/ using document root <none> and password file <none>
I20260926 13:40:26.490401 8956 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:26.490465 8956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:26.490641 8956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:26.492867 8956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "443412209c6341428c08bcb7aab1bc2b"
format_stamp: "Formatted at 2026-09-26 13:40:26 on dist-test-slave-1wns"
server_key: "ea4c4186e584e2efb0b45de37ebe5940"
server_key_iv: "5fda7f73b20781476c33e2dfd20c1d2d"
server_key_version: "encryptionkey@0"
I20260926 13:40:26.493418 8956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "443412209c6341428c08bcb7aab1bc2b"
format_stamp: "Formatted at 2026-09-26 13:40:26 on dist-test-slave-1wns"
server_key: "ea4c4186e584e2efb0b45de37ebe5940"
server_key_iv: "5fda7f73b20781476c33e2dfd20c1d2d"
server_key_version: "encryptionkey@0"
I20260926 13:40:26.497723 8956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:40:26.500489 8971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:26.501960 8956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:26.502136 8956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "443412209c6341428c08bcb7aab1bc2b"
format_stamp: "Formatted at 2026-09-26 13:40:26 on dist-test-slave-1wns"
server_key: "ea4c4186e584e2efb0b45de37ebe5940"
server_key_iv: "5fda7f73b20781476c33e2dfd20c1d2d"
server_key_version: "encryptionkey@0"
I20260926 13:40:26.502272 8956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:26.534376 8956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:26.538437 8956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:26.538769 8956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:26.540693 8956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:26.540766 8956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:26.540844 8956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:26.540875 8956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:26.553756 8956 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:43643
I20260926 13:40:26.553808 9086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:43643 every 8 connection(s)
Sep 26 13:40:26 dist-test-slave-1wns krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430026, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
I20260926 13:40:26.555109 8956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:40:26.560581 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.541997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:60067 (local address 127.1.49.62:41433)
0926 13:40:26.542188 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:26.542194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:26.543350 (+ 1156us) server_negotiation.cc:407] Connection header received
0926 13:40:26.545127 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:26.545131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:26.545202 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:26.545324 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:26.547884 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.549058 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.551439 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.551961 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.552179 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.555898 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:26.555903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:26.555907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:26.555945 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:26.557792 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.558553 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.558557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.558559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.558649 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.559106 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.559109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.559110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.559364 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:26.559500 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:26.560241 (+ 741us) server_negotiation.cc:299] Negotiation successful
0926 13:40:26.560394 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 13:40:26.561591 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.542582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41433 (local address 127.1.49.2:60067)
0926 13:40:26.543135 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:26.543196 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:26.544773 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:26.545775 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:26.545788 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:26.546434 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:26.547566 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.547598 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.549281 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.549285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.551195 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.551212 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.552219 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.552224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.552463 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.553432 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:26.553464 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:26.555575 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:26.557944 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:26.557950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:26.557965 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:26.558423 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:26.558796 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:26.558799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:26.558801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:26.558991 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:26.559517 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:26.559526 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:26.559964 (+ 438us) client_negotiation.cc:769] Sending connection context
0926 13:40:26.560193 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 13:40:26.560634 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":175,"threads_started":1}
I20260926 13:40:26.563172 9087 heartbeater.cc:347] Connected to a master server at 127.1.49.62:41433
I20260926 13:40:26.563361 9087 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:26.564049 9087 heartbeater.cc:511] Master 127.1.49.62:41433 requested a full tablet report, sending...
I20260926 13:40:26.564859 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 8956
I20260926 13:40:26.564996 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:40:26.565330 1220 external_mini_cluster.cc:1428] Setting key c0666baccfaec8c59a9e77c95494736a
I20260926 13:40:26.565315 8743 ts_manager.cc:194] Registered new tserver with Master: 443412209c6341428c08bcb7aab1bc2b (127.1.49.2:43643)
I20260926 13:40:26.566212 8743 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:60067
I20260926 13:40:26.574790 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.566121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:46821 (local address 127.1.49.62:41433)
0926 13:40:26.566226 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:26.566231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:26.566245 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:40:26.566446 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:26.566449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:26.566494 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:26.566601 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:26.567196 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.568077 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.569174 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.569439 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.569529 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.571047 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:26.571054 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:26.571060 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:26.571096 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:26.572892 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.573392 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.573396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.573398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.573491 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.573798 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.573803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.573805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.574053 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:26.574172 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:26.574475 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 13:40:26.574597 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:40:26.574822 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.565711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41433 (local address 127.1.49.2:46821)
0926 13:40:26.566072 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:26.566090 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:26.566315 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:26.566620 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:26.566623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:26.566882 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:26.567067 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.567076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.568227 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.568230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.569035 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.569047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.569561 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.569567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.569678 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.570395 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:26.570417 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:26.570857 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:26.573037 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:26.573041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:26.573043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:26.573271 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:26.573593 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:26.573598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:26.573600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:26.573695 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:26.574172 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:26.574176 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:26.574288 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 13:40:26.574414 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 13:40:26.574598 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":14,"thread_start_us":171,"threads_started":1}
WARNING: no policy specified for oryx/127.1.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.3.keytab.
Entry for principal oryx/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.3.keytab.
I20260926 13:40:26.596748 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41433
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.3.keytab
--principal=oryx/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:26.729933 9096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:26.730312 9096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:26.730391 9096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:26.735090 9096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:26.735173 9096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:26.735216 9096 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:40:26.735287 9096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:40:26.741218 9096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.3.keytab
--principal=oryx/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.49.62:41433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:26.742422 9096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:26.743623 9096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:26.754835 9101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:26.754895 9096 server_base.cc:1034] running on GCE node
W20260926 13:40:26.754818 9102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:26.755223 9104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:26.755703 9096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:26.756711 9096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:26.758071 9096 hybrid_clock.cc:648] HybridClock initialized: now 1790430026758028 us; error 56 us; skew 500 ppm
Sep 26 13:40:26 dist-test-slave-1wns krb5kdc[8699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430026, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:26.762019 9096 init.cc:377] Logged in from keytab as oryx/127.1.49.3@KRBTEST.COM (short username oryx)
I20260926 13:40:26.763271 9096 webserver.cc:460] Webserver started at http://127.1.49.3:40329/ using document root <none> and password file <none>
I20260926 13:40:26.764072 9096 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:26.764149 9096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:26.764408 9096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:26.767001 9096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "9d82750a1dcc469b9a341be7f21133e0"
format_stamp: "Formatted at 2026-09-26 13:40:26 on dist-test-slave-1wns"
server_key: "72248cd1761e67bd00d2e00cd3ea0e8d"
server_key_iv: "6011a9a807747a6a02d7673bfbe45b04"
server_key_version: "encryptionkey@0"
I20260926 13:40:26.767603 9096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "9d82750a1dcc469b9a341be7f21133e0"
format_stamp: "Formatted at 2026-09-26 13:40:26 on dist-test-slave-1wns"
server_key: "72248cd1761e67bd00d2e00cd3ea0e8d"
server_key_iv: "6011a9a807747a6a02d7673bfbe45b04"
server_key_version: "encryptionkey@0"
I20260926 13:40:26.772341 9096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:40:26.775435 9111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:26.777136 9096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:26.777413 9096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "9d82750a1dcc469b9a341be7f21133e0"
format_stamp: "Formatted at 2026-09-26 13:40:26 on dist-test-slave-1wns"
server_key: "72248cd1761e67bd00d2e00cd3ea0e8d"
server_key_iv: "6011a9a807747a6a02d7673bfbe45b04"
server_key_version: "encryptionkey@0"
I20260926 13:40:26.777571 9096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:26.795202 9096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:26.799265 9096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:26.799517 9096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:26.801255 9096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:26.801347 9096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:26.801424 9096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:26.801472 9096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:26.813491 9096 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:36299
I20260926 13:40:26.813522 9226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:36299 every 8 connection(s)
Sep 26 13:40:26 dist-test-slave-1wns krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430026, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
I20260926 13:40:26.814924 9096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:40:26.820273 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.802498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:53809 (local address 127.1.49.62:41433)
0926 13:40:26.802647 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:26.802651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:26.804042 (+ 1391us) server_negotiation.cc:407] Connection header received
0926 13:40:26.805776 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:26.805785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:26.805869 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:26.806032 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:26.807906 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.808936 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.810648 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.811081 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.811227 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.814863 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:26.814872 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:26.814879 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:26.814922 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:26.817142 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.817865 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.817871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.817874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.818015 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.818571 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.818576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.818578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.818818 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:26.818954 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:26.819934 (+ 980us) server_negotiation.cc:299] Negotiation successful
0926 13:40:26.820117 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 13:40:26.821156 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.803122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41433 (local address 127.1.49.3:53809)
0926 13:40:26.803744 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:26.803925 (+ 181us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:26.805458 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:26.806277 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:26.806287 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:26.806858 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:26.807665 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.807692 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.809075 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.809078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.810480 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.810491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.811344 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.811353 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.811634 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.812499 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:26.812518 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:26.814614 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:26.817292 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:26.817298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:26.817313 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:26.817665 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:26.818184 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:26.818187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:26.818189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:26.818415 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:26.818967 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:26.818973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:26.819530 (+ 557us) client_negotiation.cc:769] Sending connection context
0926 13:40:26.819708 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 13:40:26.820094 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":173,"threads_started":1}
I20260926 13:40:26.823148 9227 heartbeater.cc:347] Connected to a master server at 127.1.49.62:41433
I20260926 13:40:26.823377 9227 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:26.824107 9227 heartbeater.cc:511] Master 127.1.49.62:41433 requested a full tablet report, sending...
I20260926 13:40:26.824832 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 9096
I20260926 13:40:26.824963 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:40:26.825317 1220 external_mini_cluster.cc:1428] Setting key 580ea6fb5c344d972af8ca26f9c024a7
I20260926 13:40:26.825359 8739 ts_manager.cc:194] Registered new tserver with Master: 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299)
I20260926 13:40:26.826181 8739 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:53809
I20260926 13:40:26.827647 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:40:26.833814 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.825874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:46783 (local address 127.1.49.62:41433)
0926 13:40:26.825980 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:26.825985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:26.825999 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:40:26.826213 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:26.826216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:26.826299 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:26.826356 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:26.827092 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.828132 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.828975 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.829217 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.829333 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.830465 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:26.830471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:26.830478 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:26.830504 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:26.832088 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.832530 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.832534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.832535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.832618 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.832914 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.832918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.832920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.833107 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:26.833215 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:26.833525 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 13:40:26.833639 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:40:26.833823 9233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.825479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41433 (local address 127.1.49.3:46783)
0926 13:40:26.825835 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:26.825852 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:26.825946 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:26.826506 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:26.826508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:26.826780 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:26.826961 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.826970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.828269 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.828273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.828842 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.828851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.829339 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.829342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.829438 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.829976 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:26.829988 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:26.830314 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:26.832208 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:26.832212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:26.832214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:26.832429 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:26.832712 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:26.832716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:26.832718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:26.832820 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:26.833204 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:26.833207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:26.833312 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 13:40:26.833468 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 13:40:26.833642 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":151,"threads_started":1}
Sep 26 13:40:26 dist-test-slave-1wns krb5kdc[8699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430026, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:40:26 dist-test-slave-1wns krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:26 dist-test-slave-1wns krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:40:26.855420 8793 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:26.848127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.1.49.62:41433)
0926 13:40:26.848292 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:26.848296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:26.848310 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:40:26.848362 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:26.848365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:26.848408 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:26.848516 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:26.849402 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.849981 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.850974 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.851220 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.851349 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.855283 (+ 3934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35182: BlockingRecv error: recv got EOF from 127.0.0.1:35182 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 13:40:26 dist-test-slave-1wns krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430026, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
I20260926 13:40:26.867905 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.857608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.1.49.62:41433)
0926 13:40:26.857799 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:26.857803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:26.857889 (+ 86us) server_negotiation.cc:407] Connection header received
0926 13:40:26.858086 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:26.858089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:26.858136 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:26.858217 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:26.859087 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.859709 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.860942 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.861190 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.861384 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.864281 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:26.864286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:26.864289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:26.864328 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:26.865966 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.866417 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.866420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.866421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.866498 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:26.866902 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:26.866905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:26.866907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:26.867135 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:26.867259 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:26.867539 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 13:40:26.867648 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 13:40:26.871529 8739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:40:26.876876 8739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:40:26.894253 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.888123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.1.49.2:43643)
0926 13:40:26.888805 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:26.888819 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:26.888846 (+ 27us) server_negotiation.cc:407] Connection header received
0926 13:40:26.889098 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:26.889105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:26.889341 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:26.889503 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:26.890006 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.890932 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.892575 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.893466 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.893583 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.893784 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:26.893975 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 13:40:26.894088 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":122,"threads_started":1}
I20260926 13:40:26.894249 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.888092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:42283 (local address 127.0.0.1:37594)
0926 13:40:26.888994 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:26.889007 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:26.889100 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:26.890074 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:26.890078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:26.890099 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:26.890176 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.890182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.891525 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.891528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.892693 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.892700 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.893713 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.893716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.893815 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.893837 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:40:26.893918 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 13:40:26.894085 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":38,"thread_start_us":135,"threads_started":1}
I20260926 13:40:26.894515 9252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.888853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.1.49.1:42283)
0926 13:40:26.889564 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:26.889571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:26.889597 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:40:26.889681 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:26.889687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:26.889897 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:26.890076 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:26.890268 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.891399 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.892825 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.893586 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.894073 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.894204 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:26.894253 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 13:40:26.894355 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":89,"threads_started":1}
I20260926 13:40:26.894572 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.889427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.1.49.3:36299)
0926 13:40:26.889830 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:26.889841 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:26.889861 (+ 20us) server_negotiation.cc:407] Connection header received
0926 13:40:26.889931 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:26.889938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:26.890121 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:26.890562 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:26.890683 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.891692 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.893038 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.893645 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.893746 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.893903 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:26.894321 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 13:40:26.894435 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":111,"threads_started":1}
I20260926 13:40:26.895424 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.888905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:36299 (local address 127.0.0.1:54222)
0926 13:40:26.889224 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:26.889235 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:26.889322 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:26.890269 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:26.890272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:26.890289 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:26.890411 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.890418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.891849 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.891853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.892911 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.892920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.893962 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.893965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.894053 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.894072 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 13:40:26.894136 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 13:40:26.894204 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":97,"threads_started":1}
I20260926 13:40:26.895750 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:26.887971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:43643 (local address 127.0.0.1:59232)
0926 13:40:26.888657 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:26.888714 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:26.888858 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:26.889698 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:26.889705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:26.889729 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:26.889877 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.889883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.891044 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.891049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.892435 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:26.892446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:26.893602 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:26.893608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:26.893746 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:26.893789 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 13:40:26.893922 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 13:40:26.895630 (+ 1708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":25,"thread_start_us":108,"threads_started":1}
I20260926 13:40:26.897657 8881 tablet_service.cc:1467] Processing CreateTablet for tablet f951d4cb12c446a08913bcc3e82e1c65 (DEFAULT_TABLE table=test-table [id=35bf07e9206a4f8396354e4e708dc90e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:26.898336 9143 tablet_service.cc:1467] Processing CreateTablet for tablet f951d4cb12c446a08913bcc3e82e1c65 (DEFAULT_TABLE table=test-table [id=35bf07e9206a4f8396354e4e708dc90e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:26.899030 8881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f951d4cb12c446a08913bcc3e82e1c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:26.899274 9143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f951d4cb12c446a08913bcc3e82e1c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:26.899324 9021 tablet_service.cc:1467] Processing CreateTablet for tablet f951d4cb12c446a08913bcc3e82e1c65 (DEFAULT_TABLE table=test-table [id=35bf07e9206a4f8396354e4e708dc90e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:26.900382 9021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f951d4cb12c446a08913bcc3e82e1c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:26.908053 9256 tablet_bootstrap.cc:492] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179: Bootstrap starting.
I20260926 13:40:26.910768 9256 tablet_bootstrap.cc:654] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:26.911791 9256 log.cc:824] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:26.906821 9255 tablet_bootstrap.cc:492] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0: Bootstrap starting.
I20260926 13:40:26.912137 9257 tablet_bootstrap.cc:492] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b: Bootstrap starting.
I20260926 13:40:26.914935 9255 tablet_bootstrap.cc:654] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:26.915009 9257 tablet_bootstrap.cc:654] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:26.915911 9255 log.cc:824] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:26.915974 9257 log.cc:824] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:26.917495 9256 tablet_bootstrap.cc:492] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179: No bootstrap required, opened a new log
I20260926 13:40:26.917743 9256 ts_tablet_manager.cc:1397] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260926 13:40:26.918088 9255 tablet_bootstrap.cc:492] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0: No bootstrap required, opened a new log
I20260926 13:40:26.918097 9257 tablet_bootstrap.cc:492] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b: No bootstrap required, opened a new log
I20260926 13:40:26.918267 9255 ts_tablet_manager.cc:1397] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260926 13:40:26.918290 9257 ts_tablet_manager.cc:1397] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260926 13:40:26.921751 9257 raft_consensus.cc:348] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:26.922073 9257 raft_consensus.cc:374] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:26.922149 9257 raft_consensus.cc:729] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443412209c6341428c08bcb7aab1bc2b, State: Initialized, Role: FOLLOWER
I20260926 13:40:26.922657 9257 consensus_queue.cc:260] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:26.922821 9256 raft_consensus.cc:348] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:26.922781 9255 raft_consensus.cc:348] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:26.923158 9255 raft_consensus.cc:374] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:26.923159 9256 raft_consensus.cc:374] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:26.923241 9256 raft_consensus.cc:729] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bcd547e39d4ded8639619ceab74179, State: Initialized, Role: FOLLOWER
I20260926 13:40:26.923344 9255 raft_consensus.cc:729] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d82750a1dcc469b9a341be7f21133e0, State: Initialized, Role: FOLLOWER
I20260926 13:40:26.923620 9087 heartbeater.cc:503] Master 127.1.49.62:41433 was elected leader, sending a full tablet report...
I20260926 13:40:26.923913 9257 ts_tablet_manager.cc:1428] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260926 13:40:26.923872 9256 consensus_queue.cc:260] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:26.923872 9255 consensus_queue.cc:260] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:26.924569 8947 heartbeater.cc:503] Master 127.1.49.62:41433 was elected leader, sending a full tablet report...
I20260926 13:40:26.924994 9255 ts_tablet_manager.cc:1428] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260926 13:40:26.924994 9256 ts_tablet_manager.cc:1428] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 13:40:26.925385 9227 heartbeater.cc:503] Master 127.1.49.62:41433 was elected leader, sending a full tablet report...
W20260926 13:40:27.053702 8948 tablet.cc:2369] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:40:27.056655 9088 tablet.cc:2369] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:40:27.066277 9228 tablet.cc:2369] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:40:27.175730 9261 raft_consensus.cc:482] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:40:27.176070 9261 raft_consensus.cc:504] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.177510 9261 leader_election.cc:288] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50bcd547e39d4ded8639619ceab74179 (127.1.49.1:42283), 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299)
I20260926 13:40:27.181587 9252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.177919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:42489 (local address 127.1.49.1:42283)
0926 13:40:27.178059 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:27.178064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:27.178213 (+ 149us) server_negotiation.cc:407] Connection header received
0926 13:40:27.178379 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:27.178382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:27.178446 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:27.178538 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:27.178806 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.179337 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.180549 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.180988 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.181115 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.181184 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:27.181366 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 13:40:27.181445 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:40:27.181609 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.177764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:42283 (local address 127.1.49.2:42489)
0926 13:40:27.178110 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:27.178130 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:27.178264 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:27.178551 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:27.178554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:27.178571 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:27.178694 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.178700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.179455 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.179462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.180421 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.180432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.181135 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.181138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.181233 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.181261 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 13:40:27.181351 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 13:40:27.181441 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":19,"thread_start_us":92,"threads_started":1}
I20260926 13:40:27.182066 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.177785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:36299 (local address 127.1.49.2:54793)
0926 13:40:27.178266 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:27.178278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:27.178407 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:27.178698 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:27.178702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:27.178719 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:27.178855 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.178860 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.179675 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.179679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.180690 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.180700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.181528 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.181532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.181641 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.181669 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 13:40:27.181821 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 13:40:27.181902 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":19,"thread_start_us":88,"threads_started":1}
I20260926 13:40:27.182317 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.177972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:54793 (local address 127.1.49.3:36299)
0926 13:40:27.178166 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:27.178171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:27.178361 (+ 190us) server_negotiation.cc:407] Connection header received
0926 13:40:27.178528 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:27.178532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:27.178589 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:27.178684 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:27.178974 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.179550 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.180842 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.181392 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.181526 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.181592 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:27.182054 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 13:40:27.182162 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 13:40:27.182379 8901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f951d4cb12c446a08913bcc3e82e1c65" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bcd547e39d4ded8639619ceab74179" is_pre_election: true
I20260926 13:40:27.182744 8901 raft_consensus.cc:2454] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 0.
I20260926 13:40:27.182934 9180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f951d4cb12c446a08913bcc3e82e1c65" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d82750a1dcc469b9a341be7f21133e0" is_pre_election: true
I20260926 13:40:27.183255 9180 raft_consensus.cc:2454] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 0.
I20260926 13:40:27.183350 8975 leader_election.cc:302] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443412209c6341428c08bcb7aab1bc2b, 50bcd547e39d4ded8639619ceab74179; no voters:
I20260926 13:40:27.183676 9261 raft_consensus.cc:2781] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:40:27.183774 9261 raft_consensus.cc:482] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:40:27.183841 9261 raft_consensus.cc:3037] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:27.184912 9261 raft_consensus.cc:504] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.185261 9261 leader_election.cc:288] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 election: Requested vote from peers 50bcd547e39d4ded8639619ceab74179 (127.1.49.1:42283), 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299)
I20260926 13:40:27.185658 9180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f951d4cb12c446a08913bcc3e82e1c65" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d82750a1dcc469b9a341be7f21133e0"
I20260926 13:40:27.185797 9180 raft_consensus.cc:3037] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:27.185850 8901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f951d4cb12c446a08913bcc3e82e1c65" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bcd547e39d4ded8639619ceab74179"
I20260926 13:40:27.185997 8901 raft_consensus.cc:3037] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:27.187201 9180 raft_consensus.cc:2454] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 1.
I20260926 13:40:27.187217 8901 raft_consensus.cc:2454] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 1.
I20260926 13:40:27.187597 8972 leader_election.cc:302] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443412209c6341428c08bcb7aab1bc2b, 9d82750a1dcc469b9a341be7f21133e0; no voters:
I20260926 13:40:27.187866 9261 raft_consensus.cc:2781] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:27.188118 9261 raft_consensus.cc:686] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [term 1 LEADER]: Becoming Leader. State: Replica: 443412209c6341428c08bcb7aab1bc2b, State: Running, Role: LEADER
I20260926 13:40:27.188555 9261 consensus_queue.cc:237] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.192135 8744 catalog_manager.cc:5495] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 443412209c6341428c08bcb7aab1bc2b (127.1.49.2). New cstate: current_term: 1 leader_uuid: "443412209c6341428c08bcb7aab1bc2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } health_report { overall_health: UNKNOWN } } }
I20260926 13:40:27.264518 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.260426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.1.49.2:43643)
0926 13:40:27.260573 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:27.260580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:27.260597 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:40:27.260711 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:27.260718 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:27.260777 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:27.260901 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:40:27.261160 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.261907 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.262846 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.263075 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.263189 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.263481 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:40:27.264060 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:40:27.264188 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:27.264265 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 13:40:27.264327 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 13:40:27.271487 8901 raft_consensus.cc:1261] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Refusing update from remote peer 443412209c6341428c08bcb7aab1bc2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:27.271476 9180 raft_consensus.cc:1261] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Refusing update from remote peer 443412209c6341428c08bcb7aab1bc2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:27.272362 9266 consensus_queue.cc:1035] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:27.272635 9261 consensus_queue.cc:1035] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:27.279637 9269 mvcc.cc:205] Tried to move back new op lower bound from 7333601391695110144 to 7333601391367024640. Current Snapshot: MvccSnapshot[applied={T|T < 7333601391695110144}]
I20260926 13:40:27.281571 9271 mvcc.cc:205] Tried to move back new op lower bound from 7333601391695110144 to 7333601391367024640. Current Snapshot: MvccSnapshot[applied={T|T < 7333601391695110144}]
I20260926 13:40:27.282833 9270 mvcc.cc:205] Tried to move back new op lower bound from 7333601391695110144 to 7333601391367024640. Current Snapshot: MvccSnapshot[applied={T|T < 7333601391695110144}]
I20260926 13:40:27.291126 8744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:27.291347 8744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:27.297997 8744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b: Sending DeleteTablet for 3 replicas of tablet f951d4cb12c446a08913bcc3e82e1c65
I20260926 13:40:27.299142 9021 tablet_service.cc:1514] Processing DeleteTablet for tablet f951d4cb12c446a08913bcc3e82e1c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:27 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:59232
I20260926 13:40:27.299160 8881 tablet_service.cc:1514] Processing DeleteTablet for tablet f951d4cb12c446a08913bcc3e82e1c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:27 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:37594
I20260926 13:40:27.300017 9279 tablet_replica.cc:330] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179: stopping tablet replica
I20260926 13:40:27.300251 9143 tablet_service.cc:1514] Processing DeleteTablet for tablet f951d4cb12c446a08913bcc3e82e1c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:27 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:54222
I20260926 13:40:27.300421 9279 raft_consensus.cc:2229] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:27.300818 9279 raft_consensus.cc:2258] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:27.299793 9278 tablet_replica.cc:330] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b: stopping tablet replica
I20260926 13:40:27.301386 9278 raft_consensus.cc:2229] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:40:27.301826 9278 raft_consensus.cc:2258] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:27.303431 9278 ts_tablet_manager.cc:1905] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:27.305735 9280 tablet_replica.cc:330] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0: stopping tablet replica
I20260926 13:40:27.306134 9280 raft_consensus.cc:2229] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:27.306205 8744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 13:40:27.306636 9280 raft_consensus.cc:2258] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 13:40:27.306744 8744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:40:27.306829 9279 ts_tablet_manager.cc:1905] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:27.308724 9278 ts_tablet_manager.cc:1918] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:27.308881 9278 log.cc:1196] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/f951d4cb12c446a08913bcc3e82e1c65
I20260926 13:40:27.309234 9278 ts_tablet_manager.cc:1939] T f951d4cb12c446a08913bcc3e82e1c65 P 443412209c6341428c08bcb7aab1bc2b: Deleting consensus metadata
I20260926 13:40:27.310160 9279 ts_tablet_manager.cc:1918] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:27.310289 9279 log.cc:1196] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/wal/wals/f951d4cb12c446a08913bcc3e82e1c65
I20260926 13:40:27.310802 8730 catalog_manager.cc:4841] TS 443412209c6341428c08bcb7aab1bc2b (127.1.49.2:43643): tablet f951d4cb12c446a08913bcc3e82e1c65 (table test-table [id=35bf07e9206a4f8396354e4e708dc90e]) successfully deleted
I20260926 13:40:27.311767 9279 ts_tablet_manager.cc:1939] T f951d4cb12c446a08913bcc3e82e1c65 P 50bcd547e39d4ded8639619ceab74179: Deleting consensus metadata
I20260926 13:40:27.313356 8731 catalog_manager.cc:4841] TS 50bcd547e39d4ded8639619ceab74179 (127.1.49.1:42283): tablet f951d4cb12c446a08913bcc3e82e1c65 (table test-table [id=35bf07e9206a4f8396354e4e708dc90e]) successfully deleted
I20260926 13:40:27.313963 8881 tablet_service.cc:1467] Processing CreateTablet for tablet f385bc5ea00142cba019cd1ecade6172 (DEFAULT_TABLE table=test-table [id=816d6572373e427fbc0820c28eabc2c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:27.314280 8881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f385bc5ea00142cba019cd1ecade6172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:27.314390 9143 tablet_service.cc:1467] Processing CreateTablet for tablet f385bc5ea00142cba019cd1ecade6172 (DEFAULT_TABLE table=test-table [id=816d6572373e427fbc0820c28eabc2c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:27.314682 9143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f385bc5ea00142cba019cd1ecade6172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:27.317090 9256 tablet_bootstrap.cc:492] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179: Bootstrap starting.
I20260926 13:40:27.317345 9021 tablet_service.cc:1467] Processing CreateTablet for tablet f385bc5ea00142cba019cd1ecade6172 (DEFAULT_TABLE table=test-table [id=816d6572373e427fbc0820c28eabc2c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:27.317672 9021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f385bc5ea00142cba019cd1ecade6172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:27.318333 9256 tablet_bootstrap.cc:654] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:27.321070 9255 tablet_bootstrap.cc:492] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0: Bootstrap starting.
I20260926 13:40:27.322340 9255 tablet_bootstrap.cc:654] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:27.322784 9280 ts_tablet_manager.cc:1905] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:27.323098 9257 tablet_bootstrap.cc:492] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b: Bootstrap starting.
I20260926 13:40:27.324760 9257 tablet_bootstrap.cc:654] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:27.326123 9280 ts_tablet_manager.cc:1918] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:27.326260 9280 log.cc:1196] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/f951d4cb12c446a08913bcc3e82e1c65
I20260926 13:40:27.326644 9280 ts_tablet_manager.cc:1939] T f951d4cb12c446a08913bcc3e82e1c65 P 9d82750a1dcc469b9a341be7f21133e0: Deleting consensus metadata
I20260926 13:40:27.327102 9255 tablet_bootstrap.cc:492] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0: No bootstrap required, opened a new log
I20260926 13:40:27.327214 9255 ts_tablet_manager.cc:1397] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260926 13:40:27.327533 9256 tablet_bootstrap.cc:492] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179: No bootstrap required, opened a new log
I20260926 13:40:27.327620 9256 ts_tablet_manager.cc:1397] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260926 13:40:27.327708 9255 raft_consensus.cc:348] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.327864 9255 raft_consensus.cc:374] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:27.327904 9255 raft_consensus.cc:729] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d82750a1dcc469b9a341be7f21133e0, State: Initialized, Role: FOLLOWER
I20260926 13:40:27.328043 9255 consensus_queue.cc:260] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.328082 9256 raft_consensus.cc:348] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.328188 9256 raft_consensus.cc:374] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:27.328214 9256 raft_consensus.cc:729] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bcd547e39d4ded8639619ceab74179, State: Initialized, Role: FOLLOWER
I20260926 13:40:27.328236 8728 catalog_manager.cc:4841] TS 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299): tablet f951d4cb12c446a08913bcc3e82e1c65 (table test-table [id=35bf07e9206a4f8396354e4e708dc90e]) successfully deleted
I20260926 13:40:27.328375 9255 ts_tablet_manager.cc:1428] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:27.328339 9256 consensus_queue.cc:260] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.328593 9256 ts_tablet_manager.cc:1428] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:27.329797 9257 tablet_bootstrap.cc:492] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b: No bootstrap required, opened a new log
I20260926 13:40:27.329913 9257 ts_tablet_manager.cc:1397] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260926 13:40:27.330372 9257 raft_consensus.cc:348] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.330497 9257 raft_consensus.cc:374] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:27.330533 9257 raft_consensus.cc:729] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443412209c6341428c08bcb7aab1bc2b, State: Initialized, Role: FOLLOWER
I20260926 13:40:27.330659 9257 consensus_queue.cc:260] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.330930 9257 ts_tablet_manager.cc:1428] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:40:27.483331 9266 raft_consensus.cc:482] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:40:27.483520 9266 raft_consensus.cc:504] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.484091 9266 leader_election.cc:288] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50bcd547e39d4ded8639619ceab74179 (127.1.49.1:42283), 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299)
I20260926 13:40:27.484491 8901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f385bc5ea00142cba019cd1ecade6172" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bcd547e39d4ded8639619ceab74179" is_pre_election: true
I20260926 13:40:27.484516 9180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f385bc5ea00142cba019cd1ecade6172" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d82750a1dcc469b9a341be7f21133e0" is_pre_election: true
I20260926 13:40:27.484664 8901 raft_consensus.cc:2454] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 0.
I20260926 13:40:27.484674 9180 raft_consensus.cc:2454] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 0.
I20260926 13:40:27.484967 8975 leader_election.cc:302] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443412209c6341428c08bcb7aab1bc2b, 50bcd547e39d4ded8639619ceab74179; no voters:
I20260926 13:40:27.485163 9266 raft_consensus.cc:2781] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:40:27.485229 9266 raft_consensus.cc:482] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:40:27.485280 9266 raft_consensus.cc:3037] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:27.486320 9266 raft_consensus.cc:504] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.486732 9266 leader_election.cc:288] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 election: Requested vote from peers 50bcd547e39d4ded8639619ceab74179 (127.1.49.1:42283), 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299)
I20260926 13:40:27.487169 8901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f385bc5ea00142cba019cd1ecade6172" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bcd547e39d4ded8639619ceab74179"
I20260926 13:40:27.487298 8901 raft_consensus.cc:3037] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:27.487236 9180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f385bc5ea00142cba019cd1ecade6172" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d82750a1dcc469b9a341be7f21133e0"
I20260926 13:40:27.487426 9180 raft_consensus.cc:3037] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:27.488348 8901 raft_consensus.cc:2454] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 1.
I20260926 13:40:27.488621 9180 raft_consensus.cc:2454] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 1.
I20260926 13:40:27.488695 8975 leader_election.cc:302] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443412209c6341428c08bcb7aab1bc2b, 50bcd547e39d4ded8639619ceab74179; no voters:
I20260926 13:40:27.488986 9266 raft_consensus.cc:2781] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:27.489106 9266 raft_consensus.cc:686] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [term 1 LEADER]: Becoming Leader. State: Replica: 443412209c6341428c08bcb7aab1bc2b, State: Running, Role: LEADER
I20260926 13:40:27.489279 9266 consensus_queue.cc:237] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.491305 8744 catalog_manager.cc:5495] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 443412209c6341428c08bcb7aab1bc2b (127.1.49.2). New cstate: current_term: 1 leader_uuid: "443412209c6341428c08bcb7aab1bc2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } health_report { overall_health: UNKNOWN } } }
Sep 26 13:40:27 dist-test-slave-1wns krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430025, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.62@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
I20260926 13:40:27.560408 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.549979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35198 (local address 127.1.49.62:41433)
0926 13:40:27.550348 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:27.550359 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:27.550519 (+ 160us) server_negotiation.cc:407] Connection header received
0926 13:40:27.550685 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:27.550693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:27.550733 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:27.550837 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:27.551602 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.552290 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.553199 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.553458 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.553582 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.556376 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:27.556380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:27.556382 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:27.556405 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:27.558273 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:27.558907 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:27.558911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:27.558912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:27.559003 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:27.559309 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:27.559312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:27.559314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:27.559520 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:27.559639 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:27.560023 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 13:40:27.560262 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":316,"thread_start_us":118,"threads_started":1}
I20260926 13:40:27.560410 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.549932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41433 (local address 127.0.0.1:35198)
0926 13:40:27.550362 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:27.550380 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:27.550551 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:27.550858 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:27.550861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:27.551361 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:27.551463 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.551471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.552427 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.552432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.553067 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.553075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.553586 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.553590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.553666 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.554383 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:27.554397 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:27.556198 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:27.558437 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:27.558450 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:27.558457 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:27.558782 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:27.559104 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:27.559109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:27.559112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:27.559204 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:27.559637 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:27.559646 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:27.559777 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 13:40:27.559973 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 13:40:27.560190 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":146,"threads_started":1}
I20260926 13:40:27.565977 8744 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:35198:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 13:40:27.566787 8744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:40:27.572356 8881 tablet_service.cc:1467] Processing CreateTablet for tablet 164b3e808f1a4f29803ed2f1340cd5e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bed4f7d64a46400995d4325307c9cdc4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:40:27.572356 9143 tablet_service.cc:1467] Processing CreateTablet for tablet 164b3e808f1a4f29803ed2f1340cd5e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bed4f7d64a46400995d4325307c9cdc4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:40:27.572778 8881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164b3e808f1a4f29803ed2f1340cd5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:27.572778 9143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164b3e808f1a4f29803ed2f1340cd5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:27.575946 9256 tablet_bootstrap.cc:492] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179: Bootstrap starting.
I20260926 13:40:27.576128 9255 tablet_bootstrap.cc:492] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0: Bootstrap starting.
I20260926 13:40:27.576370 9021 tablet_service.cc:1467] Processing CreateTablet for tablet 164b3e808f1a4f29803ed2f1340cd5e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bed4f7d64a46400995d4325307c9cdc4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:40:27.576726 9021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164b3e808f1a4f29803ed2f1340cd5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:27.577185 9255 tablet_bootstrap.cc:654] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:27.577199 9256 tablet_bootstrap.cc:654] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:27.579586 9257 tablet_bootstrap.cc:492] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b: Bootstrap starting.
I20260926 13:40:27.580140 9255 tablet_bootstrap.cc:492] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0: No bootstrap required, opened a new log
I20260926 13:40:27.580253 9255 ts_tablet_manager.cc:1397] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 13:40:27.580318 9256 tablet_bootstrap.cc:492] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179: No bootstrap required, opened a new log
I20260926 13:40:27.580423 9256 ts_tablet_manager.cc:1397] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 13:40:27.580644 9257 tablet_bootstrap.cc:654] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:27.580792 9255 raft_consensus.cc:348] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.580945 9255 raft_consensus.cc:374] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:27.580900 9256 raft_consensus.cc:348] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.580996 9255 raft_consensus.cc:729] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d82750a1dcc469b9a341be7f21133e0, State: Initialized, Role: FOLLOWER
I20260926 13:40:27.581049 9256 raft_consensus.cc:374] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:27.581089 9256 raft_consensus.cc:729] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bcd547e39d4ded8639619ceab74179, State: Initialized, Role: FOLLOWER
I20260926 13:40:27.581213 9255 consensus_queue.cc:260] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.581238 9256 consensus_queue.cc:260] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.581561 9255 ts_tablet_manager.cc:1428] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:27.581518 9262 tablet_replica.cc:439] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } } }
I20260926 13:40:27.581523 9263 tablet_replica.cc:439] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } } }
I20260926 13:40:27.581650 9262 tablet_replica.cc:442] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:40:27.581660 9263 tablet_replica.cc:442] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:40:27.581892 9256 ts_tablet_manager.cc:1428] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:27.582511 9257 tablet_bootstrap.cc:492] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b: No bootstrap required, opened a new log
I20260926 13:40:27.582612 9257 ts_tablet_manager.cc:1397] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:40:27.582995 9257 raft_consensus.cc:348] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.583120 9257 raft_consensus.cc:374] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:27.583160 9257 raft_consensus.cc:729] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443412209c6341428c08bcb7aab1bc2b, State: Initialized, Role: FOLLOWER
I20260926 13:40:27.583312 9257 consensus_queue.cc:260] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.583573 9266 tablet_replica.cc:439] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } } }
I20260926 13:40:27.583705 9266 tablet_replica.cc:442] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:40:27.584208 9257 ts_tablet_manager.cc:1428] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:40:27.641649 9262 raft_consensus.cc:482] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:40:27.641819 9262 raft_consensus.cc:504] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.643114 9262 leader_election.cc:288] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443412209c6341428c08bcb7aab1bc2b (127.1.49.2:43643), 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299)
I20260926 13:40:27.649034 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.643595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:49463 (local address 127.1.49.2:43643)
0926 13:40:27.643935 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:27.643940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:27.643958 (+ 18us) server_negotiation.cc:407] Connection header received
0926 13:40:27.644095 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:27.644099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:27.644154 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:27.644250 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:27.644644 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.645508 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.647655 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.648312 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.648441 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.648525 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:27.648845 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 13:40:27.648928 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260926 13:40:27.649039 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.643569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:40091 (local address 127.1.49.3:36299)
0926 13:40:27.643710 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:27.643716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:27.643926 (+ 210us) server_negotiation.cc:407] Connection header received
0926 13:40:27.644095 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:27.644099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:27.644154 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:27.644250 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:27.644650 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.645560 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.647655 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.648289 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.648462 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.648520 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:27.648848 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 13:40:27.648925 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:40:27.649080 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.643429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:43643 (local address 127.1.49.1:49463)
0926 13:40:27.643784 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:27.643830 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:27.643968 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:27.644272 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:27.644275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:27.644299 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:27.644485 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.644496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.645670 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.645674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.647491 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.647503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.648456 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.648460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.648663 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.648696 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 13:40:27.648777 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 13:40:27.648897 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":341,"spinlock_wait_cycles":14976,"thread_start_us":133,"threads_started":1}
I20260926 13:40:27.649408 9295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.643423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:36299 (local address 127.1.49.1:40091)
0926 13:40:27.643784 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:27.643825 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:27.643968 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:27.644272 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:27.644275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:27.644299 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:27.644528 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.644538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.645670 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.645674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.647492 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.647503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.648458 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.648460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.648634 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.648692 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 13:40:27.649192 (+ 500us) client_negotiation.cc:240] Negotiation successful
0926 13:40:27.649262 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":367,"spinlock_wait_cycles":21248,"thread_start_us":129,"threads_started":1}
I20260926 13:40:27.649755 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164b3e808f1a4f29803ed2f1340cd5e2" candidate_uuid: "50bcd547e39d4ded8639619ceab74179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443412209c6341428c08bcb7aab1bc2b" is_pre_election: true
I20260926 13:40:27.649932 9180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164b3e808f1a4f29803ed2f1340cd5e2" candidate_uuid: "50bcd547e39d4ded8639619ceab74179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d82750a1dcc469b9a341be7f21133e0" is_pre_election: true
I20260926 13:40:27.650034 9041 raft_consensus.cc:2454] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50bcd547e39d4ded8639619ceab74179 in term 0.
I20260926 13:40:27.650111 9180 raft_consensus.cc:2454] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50bcd547e39d4ded8639619ceab74179 in term 0.
I20260926 13:40:27.650518 8834 leader_election.cc:302] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 443412209c6341428c08bcb7aab1bc2b, 50bcd547e39d4ded8639619ceab74179, 9d82750a1dcc469b9a341be7f21133e0; no voters:
I20260926 13:40:27.650816 9262 raft_consensus.cc:2781] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:40:27.650879 9262 raft_consensus.cc:482] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:40:27.650918 9262 raft_consensus.cc:3037] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:27.651961 9262 raft_consensus.cc:504] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.652339 9262 leader_election.cc:288] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [CANDIDATE]: Term 1 election: Requested vote from peers 443412209c6341428c08bcb7aab1bc2b (127.1.49.2:43643), 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299)
I20260926 13:40:27.652769 9041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164b3e808f1a4f29803ed2f1340cd5e2" candidate_uuid: "50bcd547e39d4ded8639619ceab74179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443412209c6341428c08bcb7aab1bc2b"
I20260926 13:40:27.652912 9041 raft_consensus.cc:3037] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:27.652999 9180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164b3e808f1a4f29803ed2f1340cd5e2" candidate_uuid: "50bcd547e39d4ded8639619ceab74179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d82750a1dcc469b9a341be7f21133e0"
I20260926 13:40:27.653175 9180 raft_consensus.cc:3037] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:27.654021 9041 raft_consensus.cc:2454] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50bcd547e39d4ded8639619ceab74179 in term 1.
I20260926 13:40:27.654306 9180 raft_consensus.cc:2454] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50bcd547e39d4ded8639619ceab74179 in term 1.
I20260926 13:40:27.654366 8834 leader_election.cc:302] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443412209c6341428c08bcb7aab1bc2b, 50bcd547e39d4ded8639619ceab74179; no voters:
I20260926 13:40:27.654553 9262 raft_consensus.cc:2781] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:27.654774 9262 raft_consensus.cc:686] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [term 1 LEADER]: Becoming Leader. State: Replica: 50bcd547e39d4ded8639619ceab74179, State: Running, Role: LEADER
I20260926 13:40:27.655079 9262 consensus_queue.cc:237] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.656611 9297 tablet_replica.cc:439] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179: TxnStatusTablet state changed. Reason: New leader 50bcd547e39d4ded8639619ceab74179. Latest consensus state: current_term: 1 leader_uuid: "50bcd547e39d4ded8639619ceab74179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } } }
I20260926 13:40:27.656744 9297 tablet_replica.cc:442] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:40:27.657078 9299 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 13:40:27.657238 9299 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 13:40:27.657881 8743 catalog_manager.cc:5495] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50bcd547e39d4ded8639619ceab74179 (127.1.49.1). New cstate: current_term: 1 leader_uuid: "50bcd547e39d4ded8639619ceab74179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } health_report { overall_health: UNKNOWN } } }
I20260926 13:40:27.686066 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.681750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:42283 (local address 127.0.0.1:37598)
0926 13:40:27.681939 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:27.681954 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:27.682092 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:27.682664 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:27.682667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:27.682676 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:40:27.682829 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.682836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.683683 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.683686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.684377 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.684386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.684872 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.684875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.684968 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.685013 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:40:27.685763 (+ 750us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:40:27.685779 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 13:40:27.685847 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 13:40:27.685937 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 13:40:27.686100 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.681937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.1.49.1:42283)
0926 13:40:27.682362 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:27.682369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:27.682381 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:40:27.682456 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:27.682458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:27.682511 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:27.682639 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:40:27.682943 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.683554 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.684516 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.684750 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.684859 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.685170 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:40:27.685647 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:40:27.685775 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:27.685857 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 13:40:27.685913 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":144,"threads_started":1}
I20260926 13:40:27.689313 9180 raft_consensus.cc:1261] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Refusing update from remote peer 50bcd547e39d4ded8639619ceab74179: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:27.689634 9041 raft_consensus.cc:1261] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b [term 1 FOLLOWER]: Refusing update from remote peer 50bcd547e39d4ded8639619ceab74179: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:27.689847 9262 consensus_queue.cc:1035] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:27.690241 9297 consensus_queue.cc:1035] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:27.691347 9263 tablet_replica.cc:439] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0: TxnStatusTablet state changed. Reason: New leader 50bcd547e39d4ded8639619ceab74179. Latest consensus state: current_term: 1 leader_uuid: "50bcd547e39d4ded8639619ceab74179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } } }
I20260926 13:40:27.691468 9263 tablet_replica.cc:442] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:40:27.692106 9266 tablet_replica.cc:439] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b: TxnStatusTablet state changed. Reason: New leader 50bcd547e39d4ded8639619ceab74179. Latest consensus state: current_term: 1 leader_uuid: "50bcd547e39d4ded8639619ceab74179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } } }
I20260926 13:40:27.692226 9266 tablet_replica.cc:442] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:40:27.693995 9262 tablet_replica.cc:439] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50bcd547e39d4ded8639619ceab74179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } } }
I20260926 13:40:27.694121 9262 tablet_replica.cc:442] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:40:27.695304 9263 tablet_replica.cc:439] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50bcd547e39d4ded8639619ceab74179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } } }
I20260926 13:40:27.695446 9263 tablet_replica.cc:442] T 164b3e808f1a4f29803ed2f1340cd5e2 P 9d82750a1dcc469b9a341be7f21133e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:40:27.704700 9262 tablet_replica.cc:439] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50bcd547e39d4ded8639619ceab74179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } } }
I20260926 13:40:27.704885 9262 tablet_replica.cc:442] T 164b3e808f1a4f29803ed2f1340cd5e2 P 50bcd547e39d4ded8639619ceab74179: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:40:27.706821 9266 tablet_replica.cc:439] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50bcd547e39d4ded8639619ceab74179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } } }
I20260926 13:40:27.706979 9266 tablet_replica.cc:442] T 164b3e808f1a4f29803ed2f1340cd5e2 P 443412209c6341428c08bcb7aab1bc2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:40:27.712020 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.708023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:50997 (local address 127.1.49.1:42283)
0926 13:40:27.708137 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:27.708141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:27.708397 (+ 256us) server_negotiation.cc:407] Connection header received
0926 13:40:27.708547 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:27.708550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:27.708602 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:27.708682 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:40:27.708907 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.709701 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.710683 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.711041 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.711187 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.711520 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:40:27.711611 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:40:27.711684 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:27.711782 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 13:40:27.711872 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:40:27.712038 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.707783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:42283 (local address 127.1.49.2:50997)
0926 13:40:27.708300 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:27.708312 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:27.708441 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:27.708709 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:27.708712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:27.708721 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:40:27.708822 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.708827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.709837 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.709840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.710560 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.710570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.711188 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.711191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.711329 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.711382 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:40:27.711698 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:40:27.711702 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:40:27.711762 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 13:40:27.711909 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":134,"threads_started":1}
I20260926 13:40:27.720228 8901 raft_consensus.cc:1261] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Refusing update from remote peer 443412209c6341428c08bcb7aab1bc2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:27.720450 9180 raft_consensus.cc:1261] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Refusing update from remote peer 443412209c6341428c08bcb7aab1bc2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:27.720852 9266 consensus_queue.cc:1035] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:27.721082 9261 consensus_queue.cc:1035] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:27.746392 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.740174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:52171 (local address 127.1.49.2:43643)
0926 13:40:27.740357 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:27.740363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:27.741065 (+ 702us) server_negotiation.cc:407] Connection header received
0926 13:40:27.741552 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:27.741555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:27.741613 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:27.741691 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:40:27.742251 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.743056 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.744536 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.744906 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.745051 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.745807 (+ 756us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:40:27.745901 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:40:27.745975 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:27.746181 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 13:40:27.746247 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:40:27.746596 9313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.739713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:43643 (local address 127.1.49.1:52171)
0926 13:40:27.740652 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:27.740993 (+ 341us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:27.741407 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:27.741981 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:27.741984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:27.741992 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:40:27.742125 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.742132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.743355 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.743358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.744392 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:27.744407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.745055 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.745060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:27.745542 (+ 482us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.745618 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:40:27.746003 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:40:27.746010 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:40:27.746145 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 13:40:27.746243 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":168,"threads_started":1}
I20260926 13:40:27.771690 8743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:27.771906 8743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:27.774199 8743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b: Sending DeleteTablet for 3 replicas of tablet f385bc5ea00142cba019cd1ecade6172
I20260926 13:40:27.774850 8881 tablet_service.cc:1514] Processing DeleteTablet for tablet f385bc5ea00142cba019cd1ecade6172 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:27 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:37594
I20260926 13:40:27.774889 9021 tablet_service.cc:1514] Processing DeleteTablet for tablet f385bc5ea00142cba019cd1ecade6172 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:27 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:59232
I20260926 13:40:27.775050 9143 tablet_service.cc:1514] Processing DeleteTablet for tablet f385bc5ea00142cba019cd1ecade6172 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:27 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:54222
I20260926 13:40:27.775094 9278 tablet_replica.cc:330] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b: stopping tablet replica
I20260926 13:40:27.775112 9279 tablet_replica.cc:330] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179: stopping tablet replica
I20260926 13:40:27.775214 9280 tablet_replica.cc:330] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0: stopping tablet replica
I20260926 13:40:27.775230 9278 raft_consensus.cc:2229] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:40:27.775233 9279 raft_consensus.cc:2229] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:27.775303 9280 raft_consensus.cc:2229] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:27.775359 9279 raft_consensus.cc:2258] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:27.775408 9280 raft_consensus.cc:2258] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:27.775439 9278 raft_consensus.cc:2258] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:27.776280 9278 ts_tablet_manager.cc:1905] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:27.776295 9280 ts_tablet_manager.cc:1905] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:27.776443 9279 ts_tablet_manager.cc:1905] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:27.779285 9278 ts_tablet_manager.cc:1918] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:40:27.779372 9278 log.cc:1196] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/f385bc5ea00142cba019cd1ecade6172
I20260926 13:40:27.779618 9278 ts_tablet_manager.cc:1939] T f385bc5ea00142cba019cd1ecade6172 P 443412209c6341428c08bcb7aab1bc2b: Deleting consensus metadata
I20260926 13:40:27.779947 9280 ts_tablet_manager.cc:1918] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:40:27.780104 9280 log.cc:1196] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/f385bc5ea00142cba019cd1ecade6172
I20260926 13:40:27.780431 9280 ts_tablet_manager.cc:1939] T f385bc5ea00142cba019cd1ecade6172 P 9d82750a1dcc469b9a341be7f21133e0: Deleting consensus metadata
I20260926 13:40:27.781461 8728 catalog_manager.cc:4841] TS 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299): tablet f385bc5ea00142cba019cd1ecade6172 (table test-table [id=816d6572373e427fbc0820c28eabc2c8]) successfully deleted
I20260926 13:40:27.782981 9279 ts_tablet_manager.cc:1918] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:40:27.783073 9279 log.cc:1196] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/wal/wals/f385bc5ea00142cba019cd1ecade6172
I20260926 13:40:27.783069 8730 catalog_manager.cc:4841] TS 443412209c6341428c08bcb7aab1bc2b (127.1.49.2:43643): tablet f385bc5ea00142cba019cd1ecade6172 (table test-table [id=816d6572373e427fbc0820c28eabc2c8]) successfully deleted
I20260926 13:40:27.783340 9279 ts_tablet_manager.cc:1939] T f385bc5ea00142cba019cd1ecade6172 P 50bcd547e39d4ded8639619ceab74179: Deleting consensus metadata
I20260926 13:40:27.784080 8731 catalog_manager.cc:4841] TS 50bcd547e39d4ded8639619ceab74179 (127.1.49.1:42283): tablet f385bc5ea00142cba019cd1ecade6172 (table test-table [id=816d6572373e427fbc0820c28eabc2c8]) successfully deleted
Sep 26 13:40:27 dist-test-slave-1wns krb5kdc[8699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:40:27 dist-test-slave-1wns krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:27 dist-test-slave-1wns krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:40:27.808393 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:27.800648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.1.49.62:41433)
0926 13:40:27.800829 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:27.800835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:27.800970 (+ 135us) server_negotiation.cc:407] Connection header received
0926 13:40:27.801083 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:27.801086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:27.801147 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:27.801275 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:27.802094 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.802955 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.803924 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.804153 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.804277 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.808235 (+ 3958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35214: BlockingRecv error: recv got EOF from 127.0.0.1:35214 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 13:40:27 dist-test-slave-1wns krb5kdc[8699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM
I20260926 13:40:27.821426 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:27.810196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.1.49.62:41433)
0926 13:40:27.810337 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:27.810343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:27.810366 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:40:27.810521 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:27.810525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:27.810586 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:27.810708 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:27.811392 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.812221 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.813209 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:27.813551 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:27.813694 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:27.817146 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:27.817151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:27.817155 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:27.817194 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:27.819244 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:27.819944 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:27.819949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:27.819951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:27.820048 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:27.820400 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:27.820403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:27.820405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:27.820650 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:27.820790 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:27.821092 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 13:40:27.821241 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:40:27.825350 8743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:40:27.825878 8743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:40:27.832068 9143 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2470ace8dc4b5b9226664d271e651b (DEFAULT_TABLE table=test-table [id=19e1ada1caa94225b8781a0ce6ed1c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:27.832068 9021 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2470ace8dc4b5b9226664d271e651b (DEFAULT_TABLE table=test-table [id=19e1ada1caa94225b8781a0ce6ed1c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:27.832412 9021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2470ace8dc4b5b9226664d271e651b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:27.832412 9143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2470ace8dc4b5b9226664d271e651b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:27.835130 9257 tablet_bootstrap.cc:492] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b: Bootstrap starting.
I20260926 13:40:27.835270 8881 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2470ace8dc4b5b9226664d271e651b (DEFAULT_TABLE table=test-table [id=19e1ada1caa94225b8781a0ce6ed1c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:27.835583 8881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2470ace8dc4b5b9226664d271e651b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:27.836158 9257 tablet_bootstrap.cc:654] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:27.837410 9257 tablet_bootstrap.cc:492] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b: No bootstrap required, opened a new log
I20260926 13:40:27.837491 9257 ts_tablet_manager.cc:1397] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:40:27.837973 9257 raft_consensus.cc:348] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.838093 9257 raft_consensus.cc:374] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:27.838124 9257 raft_consensus.cc:729] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443412209c6341428c08bcb7aab1bc2b, State: Initialized, Role: FOLLOWER
I20260926 13:40:27.838274 9257 consensus_queue.cc:260] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.838557 9257 ts_tablet_manager.cc:1428] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:27.839182 9256 tablet_bootstrap.cc:492] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179: Bootstrap starting.
I20260926 13:40:27.839793 9255 tablet_bootstrap.cc:492] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0: Bootstrap starting.
I20260926 13:40:27.840118 9256 tablet_bootstrap.cc:654] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:27.840711 9255 tablet_bootstrap.cc:654] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:27.842419 9256 tablet_bootstrap.cc:492] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179: No bootstrap required, opened a new log
I20260926 13:40:27.842509 9256 ts_tablet_manager.cc:1397] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 13:40:27.842954 9256 raft_consensus.cc:348] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.843066 9256 raft_consensus.cc:374] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:27.843096 9256 raft_consensus.cc:729] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bcd547e39d4ded8639619ceab74179, State: Initialized, Role: FOLLOWER
I20260926 13:40:27.843217 9256 consensus_queue.cc:260] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.843487 9256 ts_tablet_manager.cc:1428] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:27.844892 9255 tablet_bootstrap.cc:492] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0: No bootstrap required, opened a new log
I20260926 13:40:27.844981 9255 ts_tablet_manager.cc:1397] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 13:40:27.845404 9255 raft_consensus.cc:348] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.845510 9255 raft_consensus.cc:374] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:27.845537 9255 raft_consensus.cc:729] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d82750a1dcc469b9a341be7f21133e0, State: Initialized, Role: FOLLOWER
I20260926 13:40:27.845655 9255 consensus_queue.cc:260] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:27.845882 9255 ts_tablet_manager.cc:1428] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:28.037094 9261 raft_consensus.cc:482] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:40:28.037273 9261 raft_consensus.cc:504] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:28.037729 9261 leader_election.cc:288] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50bcd547e39d4ded8639619ceab74179 (127.1.49.1:42283), 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299)
I20260926 13:40:28.038308 8901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2470ace8dc4b5b9226664d271e651b" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bcd547e39d4ded8639619ceab74179" is_pre_election: true
I20260926 13:40:28.038390 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2470ace8dc4b5b9226664d271e651b" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d82750a1dcc469b9a341be7f21133e0" is_pre_election: true
I20260926 13:40:28.038488 8901 raft_consensus.cc:2454] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 0.
I20260926 13:40:28.038518 9181 raft_consensus.cc:2454] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 0.
I20260926 13:40:28.038892 8972 leader_election.cc:302] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 443412209c6341428c08bcb7aab1bc2b, 50bcd547e39d4ded8639619ceab74179, 9d82750a1dcc469b9a341be7f21133e0; no voters:
I20260926 13:40:28.039100 9261 raft_consensus.cc:2781] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:40:28.039160 9261 raft_consensus.cc:482] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:40:28.039207 9261 raft_consensus.cc:3037] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:28.040339 9261 raft_consensus.cc:504] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:28.040700 9261 leader_election.cc:288] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 election: Requested vote from peers 50bcd547e39d4ded8639619ceab74179 (127.1.49.1:42283), 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299)
I20260926 13:40:28.041138 8901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2470ace8dc4b5b9226664d271e651b" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bcd547e39d4ded8639619ceab74179"
I20260926 13:40:28.041169 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2470ace8dc4b5b9226664d271e651b" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d82750a1dcc469b9a341be7f21133e0"
I20260926 13:40:28.041260 8901 raft_consensus.cc:3037] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:28.041329 9181 raft_consensus.cc:3037] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:28.042311 9181 raft_consensus.cc:2454] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 1.
I20260926 13:40:28.042343 8901 raft_consensus.cc:2454] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 1.
I20260926 13:40:28.042680 8972 leader_election.cc:302] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443412209c6341428c08bcb7aab1bc2b, 9d82750a1dcc469b9a341be7f21133e0; no voters:
I20260926 13:40:28.042891 9261 raft_consensus.cc:2781] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:28.042999 9261 raft_consensus.cc:686] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [term 1 LEADER]: Becoming Leader. State: Replica: 443412209c6341428c08bcb7aab1bc2b, State: Running, Role: LEADER
I20260926 13:40:28.043148 9261 consensus_queue.cc:237] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:28.045143 8743 catalog_manager.cc:5495] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 443412209c6341428c08bcb7aab1bc2b (127.1.49.2). New cstate: current_term: 1 leader_uuid: "443412209c6341428c08bcb7aab1bc2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } health_report { overall_health: UNKNOWN } } }
I20260926 13:40:28.089500 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:28.086213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59252 (local address 127.1.49.2:43643)
0926 13:40:28.086376 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:28.086380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:28.086396 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:40:28.086490 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:28.086492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:28.086542 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:28.086611 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:40:28.086948 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:28.087507 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:28.088433 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:28.088663 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:28.088755 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:28.089023 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:40:28.089096 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:40:28.089152 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:28.089293 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 13:40:28.089340 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 13:40:28.091749 8901 raft_consensus.cc:1261] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Refusing update from remote peer 443412209c6341428c08bcb7aab1bc2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:28.091749 9181 raft_consensus.cc:1261] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Refusing update from remote peer 443412209c6341428c08bcb7aab1bc2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:28.092330 9266 consensus_queue.cc:1035] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:28.092537 9261 consensus_queue.cc:1035] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:28.099992 8743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:28.100171 8743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:28.102240 8743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b: Sending DeleteTablet for 3 replicas of tablet 2a2470ace8dc4b5b9226664d271e651b
I20260926 13:40:28.102907 8881 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a2470ace8dc4b5b9226664d271e651b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:28 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:37594
I20260926 13:40:28.103008 9143 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a2470ace8dc4b5b9226664d271e651b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:28 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:54222
I20260926 13:40:28.103058 9021 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a2470ace8dc4b5b9226664d271e651b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:28 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:59232
I20260926 13:40:28.103119 9279 tablet_replica.cc:330] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179: stopping tablet replica
I20260926 13:40:28.103228 9279 raft_consensus.cc:2229] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:28.103263 9278 tablet_replica.cc:330] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b: stopping tablet replica
I20260926 13:40:28.103355 9279 raft_consensus.cc:2258] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:28.103403 9278 raft_consensus.cc:2229] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:40:28.103541 9280 tablet_replica.cc:330] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0: stopping tablet replica
I20260926 13:40:28.103614 9278 raft_consensus.cc:2258] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:28.103672 9280 raft_consensus.cc:2229] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:28.103832 9280 raft_consensus.cc:2258] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:28.103881 8743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:40:28.104475 8743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:40:28.104559 9279 ts_tablet_manager.cc:1905] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:28.104597 9278 ts_tablet_manager.cc:1905] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:28.107249 9279 ts_tablet_manager.cc:1918] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:28.107339 9279 log.cc:1196] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-0/wal/wals/2a2470ace8dc4b5b9226664d271e651b
I20260926 13:40:28.107451 9278 ts_tablet_manager.cc:1918] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:28.107512 9278 log.cc:1196] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/2a2470ace8dc4b5b9226664d271e651b
I20260926 13:40:28.107604 9279 ts_tablet_manager.cc:1939] T 2a2470ace8dc4b5b9226664d271e651b P 50bcd547e39d4ded8639619ceab74179: Deleting consensus metadata
I20260926 13:40:28.107698 9278 ts_tablet_manager.cc:1939] T 2a2470ace8dc4b5b9226664d271e651b P 443412209c6341428c08bcb7aab1bc2b: Deleting consensus metadata
I20260926 13:40:28.108673 8731 catalog_manager.cc:4841] TS 50bcd547e39d4ded8639619ceab74179 (127.1.49.1:42283): tablet 2a2470ace8dc4b5b9226664d271e651b (table test-table [id=19e1ada1caa94225b8781a0ce6ed1c05]) successfully deleted
I20260926 13:40:28.108860 8730 catalog_manager.cc:4841] TS 443412209c6341428c08bcb7aab1bc2b (127.1.49.2:43643): tablet 2a2470ace8dc4b5b9226664d271e651b (table test-table [id=19e1ada1caa94225b8781a0ce6ed1c05]) successfully deleted
I20260926 13:40:28.108891 9280 ts_tablet_manager.cc:1905] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:28.110980 9280 ts_tablet_manager.cc:1918] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:28.111048 9280 log.cc:1196] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/2a2470ace8dc4b5b9226664d271e651b
I20260926 13:40:28.111238 9280 ts_tablet_manager.cc:1939] T 2a2470ace8dc4b5b9226664d271e651b P 9d82750a1dcc469b9a341be7f21133e0: Deleting consensus metadata
I20260926 13:40:28.112305 8728 catalog_manager.cc:4841] TS 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299): tablet 2a2470ace8dc4b5b9226664d271e651b (table test-table [id=19e1ada1caa94225b8781a0ce6ed1c05]) successfully deleted
I20260926 13:40:28.112435 9021 tablet_service.cc:1467] Processing CreateTablet for tablet 753ef1d365944b9db3703e0d5e595818 (DEFAULT_TABLE table=test-table [id=1a10226a2f6f4e19a6beb3fa6f813cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:28.112632 8881 tablet_service.cc:1467] Processing CreateTablet for tablet 753ef1d365944b9db3703e0d5e595818 (DEFAULT_TABLE table=test-table [id=1a10226a2f6f4e19a6beb3fa6f813cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:28.112756 9021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753ef1d365944b9db3703e0d5e595818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:28.112954 8881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753ef1d365944b9db3703e0d5e595818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:28.114653 9257 tablet_bootstrap.cc:492] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b: Bootstrap starting.
I20260926 13:40:28.115234 9143 tablet_service.cc:1467] Processing CreateTablet for tablet 753ef1d365944b9db3703e0d5e595818 (DEFAULT_TABLE table=test-table [id=1a10226a2f6f4e19a6beb3fa6f813cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:28.115567 9143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753ef1d365944b9db3703e0d5e595818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:28.115619 9257 tablet_bootstrap.cc:654] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:28.116762 9257 tablet_bootstrap.cc:492] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b: No bootstrap required, opened a new log
I20260926 13:40:28.116842 9257 ts_tablet_manager.cc:1397] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:40:28.117324 9257 raft_consensus.cc:348] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:28.117422 9256 tablet_bootstrap.cc:492] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179: Bootstrap starting.
I20260926 13:40:28.117462 9257 raft_consensus.cc:374] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:28.117506 9257 raft_consensus.cc:729] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443412209c6341428c08bcb7aab1bc2b, State: Initialized, Role: FOLLOWER
I20260926 13:40:28.117641 9257 consensus_queue.cc:260] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:28.117942 9257 ts_tablet_manager.cc:1428] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:28.118709 9256 tablet_bootstrap.cc:654] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:28.120466 9255 tablet_bootstrap.cc:492] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0: Bootstrap starting.
I20260926 13:40:28.121052 9256 tablet_bootstrap.cc:492] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179: No bootstrap required, opened a new log
I20260926 13:40:28.121146 9256 ts_tablet_manager.cc:1397] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 13:40:28.121556 9255 tablet_bootstrap.cc:654] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:28.121628 9256 raft_consensus.cc:348] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:28.121757 9256 raft_consensus.cc:374] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:28.121798 9256 raft_consensus.cc:729] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bcd547e39d4ded8639619ceab74179, State: Initialized, Role: FOLLOWER
I20260926 13:40:28.121965 9256 consensus_queue.cc:260] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:28.122291 9256 ts_tablet_manager.cc:1428] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:40:28.122910 9255 tablet_bootstrap.cc:492] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0: No bootstrap required, opened a new log
I20260926 13:40:28.123076 9255 ts_tablet_manager.cc:1397] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:40:28.123620 9255 raft_consensus.cc:348] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:28.123772 9255 raft_consensus.cc:374] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:28.123858 9255 raft_consensus.cc:729] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d82750a1dcc469b9a341be7f21133e0, State: Initialized, Role: FOLLOWER
I20260926 13:40:28.124017 9255 consensus_queue.cc:260] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:28.124389 9255 ts_tablet_manager.cc:1428] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:40:28.405345 9334 raft_consensus.cc:482] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:40:28.405555 9334 raft_consensus.cc:504] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:28.406087 9334 leader_election.cc:288] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50bcd547e39d4ded8639619ceab74179 (127.1.49.1:42283), 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299)
I20260926 13:40:28.406684 8901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753ef1d365944b9db3703e0d5e595818" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bcd547e39d4ded8639619ceab74179" is_pre_election: true
I20260926 13:40:28.406728 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753ef1d365944b9db3703e0d5e595818" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d82750a1dcc469b9a341be7f21133e0" is_pre_election: true
I20260926 13:40:28.406873 8901 raft_consensus.cc:2454] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 0.
I20260926 13:40:28.406912 9181 raft_consensus.cc:2454] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 0.
I20260926 13:40:28.407322 8972 leader_election.cc:302] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443412209c6341428c08bcb7aab1bc2b, 9d82750a1dcc469b9a341be7f21133e0; no voters:
I20260926 13:40:28.407523 9334 raft_consensus.cc:2781] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:40:28.407585 9334 raft_consensus.cc:482] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:40:28.407644 9334 raft_consensus.cc:3037] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:28.408774 9334 raft_consensus.cc:504] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:28.409207 9334 leader_election.cc:288] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 election: Requested vote from peers 50bcd547e39d4ded8639619ceab74179 (127.1.49.1:42283), 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299)
I20260926 13:40:28.409657 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753ef1d365944b9db3703e0d5e595818" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d82750a1dcc469b9a341be7f21133e0"
I20260926 13:40:28.409657 8901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753ef1d365944b9db3703e0d5e595818" candidate_uuid: "443412209c6341428c08bcb7aab1bc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bcd547e39d4ded8639619ceab74179"
I20260926 13:40:28.409797 9181 raft_consensus.cc:3037] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:28.409797 8901 raft_consensus.cc:3037] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:28.410985 8901 raft_consensus.cc:2454] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 1.
I20260926 13:40:28.410985 9181 raft_consensus.cc:2454] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443412209c6341428c08bcb7aab1bc2b in term 1.
I20260926 13:40:28.411368 8972 leader_election.cc:302] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443412209c6341428c08bcb7aab1bc2b, 9d82750a1dcc469b9a341be7f21133e0; no voters:
I20260926 13:40:28.411582 9334 raft_consensus.cc:2781] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:28.411695 9334 raft_consensus.cc:686] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [term 1 LEADER]: Becoming Leader. State: Replica: 443412209c6341428c08bcb7aab1bc2b, State: Running, Role: LEADER
I20260926 13:40:28.411916 9334 consensus_queue.cc:237] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } }
I20260926 13:40:28.413975 8743 catalog_manager.cc:5495] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 443412209c6341428c08bcb7aab1bc2b (127.1.49.2). New cstate: current_term: 1 leader_uuid: "443412209c6341428c08bcb7aab1bc2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443412209c6341428c08bcb7aab1bc2b" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 } health_report { overall_health: UNKNOWN } } }
I20260926 13:40:28.502167 8901 raft_consensus.cc:1261] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Refusing update from remote peer 443412209c6341428c08bcb7aab1bc2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:28.502594 9181 raft_consensus.cc:1261] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Refusing update from remote peer 443412209c6341428c08bcb7aab1bc2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:28.502727 9266 consensus_queue.cc:1035] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "50bcd547e39d4ded8639619ceab74179" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:28.503021 9334 consensus_queue.cc:1035] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d82750a1dcc469b9a341be7f21133e0" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:28.537676 8744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:28.537858 8744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:28.540050 8744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c624eaa121ab4590a69af5dc5e9d5c9b: Sending DeleteTablet for 3 replicas of tablet 753ef1d365944b9db3703e0d5e595818
I20260926 13:40:28.540841 9021 tablet_service.cc:1514] Processing DeleteTablet for tablet 753ef1d365944b9db3703e0d5e595818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:28 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:59232
I20260926 13:40:28.540838 8881 tablet_service.cc:1514] Processing DeleteTablet for tablet 753ef1d365944b9db3703e0d5e595818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:28 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:37594
I20260926 13:40:28.540898 9143 tablet_service.cc:1514] Processing DeleteTablet for tablet 753ef1d365944b9db3703e0d5e595818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:28 UTC) from {username='oryx', principal='oryx/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:54222
I20260926 13:40:28.541065 9278 tablet_replica.cc:330] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b: stopping tablet replica
I20260926 13:40:28.541080 9279 tablet_replica.cc:330] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179: stopping tablet replica
I20260926 13:40:28.541208 9278 raft_consensus.cc:2229] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:40:28.541257 9279 raft_consensus.cc:2229] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:28.541378 9279 raft_consensus.cc:2258] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:28.541468 9278 raft_consensus.cc:2258] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:28.541672 9280 tablet_replica.cc:330] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0: stopping tablet replica
I20260926 13:40:28.541813 9280 raft_consensus.cc:2229] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:28.541970 9280 raft_consensus.cc:2258] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:28.542253 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 8816
I20260926 13:40:28.542446 9279 ts_tablet_manager.cc:1905] T 753ef1d365944b9db3703e0d5e595818 P 50bcd547e39d4ded8639619ceab74179: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:28.543936 9280 ts_tablet_manager.cc:1905] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:28.544215 9278 ts_tablet_manager.cc:1905] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:28.546200 9278 ts_tablet_manager.cc:1918] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:40:28.546257 9278 log.cc:1196] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/753ef1d365944b9db3703e0d5e595818
I20260926 13:40:28.546445 9278 ts_tablet_manager.cc:1939] T 753ef1d365944b9db3703e0d5e595818 P 443412209c6341428c08bcb7aab1bc2b: Deleting consensus metadata
I20260926 13:40:28.547194 9280 ts_tablet_manager.cc:1918] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:40:28.547271 9280 log.cc:1196] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/753ef1d365944b9db3703e0d5e595818
I20260926 13:40:28.547271 8730 catalog_manager.cc:4841] TS 443412209c6341428c08bcb7aab1bc2b (127.1.49.2:43643): tablet 753ef1d365944b9db3703e0d5e595818 (table test-table [id=1a10226a2f6f4e19a6beb3fa6f813cc9]) successfully deleted
I20260926 13:40:28.547526 9280 ts_tablet_manager.cc:1939] T 753ef1d365944b9db3703e0d5e595818 P 9d82750a1dcc469b9a341be7f21133e0: Deleting consensus metadata
I20260926 13:40:28.548480 8728 catalog_manager.cc:4841] TS 9d82750a1dcc469b9a341be7f21133e0 (127.1.49.3:36299): tablet 753ef1d365944b9db3703e0d5e595818 (table test-table [id=1a10226a2f6f4e19a6beb3fa6f813cc9]) successfully deleted
W20260926 13:40:28.550848 8731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:42283 (error 108)
I20260926 13:40:28.551100 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 8956
I20260926 13:40:28.553251 9346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:28.552806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:42283 (local address 127.0.0.1:37608)
0926 13:40:28.553084 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:28.553162 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
W20260926 13:40:28.553408 8731 catalog_manager.cc:4584] TS 50bcd547e39d4ded8639619ceab74179 (127.1.49.1:42283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 753ef1d365944b9db3703e0d5e595818: Network error: Client connection negotiation failed: client connection to 127.1.49.1:42283: connect: Connection refused (error 111)
I20260926 13:40:28.558386 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 9096
I20260926 13:40:28.564280 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 8712
2026-09-26T13:40:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4027 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:40:28 dist-test-slave-1wns krb5kdc[9349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:40:28 dist-test-slave-1wns krb5kdc[9349](info): set up 2 sockets
Sep 26 13:40:28 dist-test-slave-1wns krb5kdc[9349](info): commencing operation
krb5kdc: starting...
W20260926 13:40:29.680666 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:40:29 dist-test-slave-1wns krb5kdc[9349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:40:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:40:29.811075 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:32951
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:35055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:32951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:29.927686 9362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:29.928136 9362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:29.928217 9362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:29.932618 9362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:29.932730 9362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:29.932771 9362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:29.932808 9362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:29.932842 9362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:29.938825 9362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:35055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:32951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:32951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:29.940179 9362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:29.941399 9362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:29.950908 9370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:29.950968 9362 server_base.cc:1034] running on GCE node
W20260926 13:40:29.950974 9368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:29.950911 9367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:29.951998 9362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:29.953315 9362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:29.954537 9362 hybrid_clock.cc:648] HybridClock initialized: now 1790430029954495 us; error 60 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T13:40:29Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1392 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:40:30 dist-test-slave-1wns krb5kdc[9375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:40:30 dist-test-slave-1wns krb5kdc[9375](info): set up 2 sockets
Sep 26 13:40:30 dist-test-slave-1wns krb5kdc[9375](info): commencing operation
krb5kdc: starting...
W20260926 13:40:31.064864 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:40:31 dist-test-slave-1wns krb5kdc[9375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:40:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:40:31.204960 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:33833
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:33833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:31.332671 9388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:31.332990 9388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:31.333035 9388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:31.337450 9388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:31.337556 9388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:31.337579 9388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:31.337600 9388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:31.337620 9388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:31.343899 9388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:33833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:33833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:31.345162 9388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:31.346364 9388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:31.356142 9396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:31.356202 9388 server_base.cc:1034] running on GCE node
W20260926 13:40:31.356122 9394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:31.356122 9393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:31.357205 9388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:31.358517 9388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:31.359730 9388 hybrid_clock.cc:648] HybridClock initialized: now 1790430031359692 us; error 52 us; skew 500 ppm
Sep 26 13:40:31 dist-test-slave-1wns krb5kdc[9375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430031, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:31.370700 9388 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:40:31.371850 9388 webserver.cc:460] Webserver started at http://127.1.49.62:40933/ using document root <none> and password file <none>
I20260926 13:40:31.372565 9388 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:31.372639 9388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:31.372826 9388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:31.375248 9388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "e4e3ae1ba49d4c53bca332161137d7c4"
format_stamp: "Formatted at 2026-09-26 13:40:31 on dist-test-slave-1wns"
server_key: "2ce194668e74bf50d1550dac3d6c3892"
server_key_iv: "b5c42fbb5fff4fca73c2f3fb3503ee06"
server_key_version: "encryptionkey@0"
I20260926 13:40:31.375921 9388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "e4e3ae1ba49d4c53bca332161137d7c4"
format_stamp: "Formatted at 2026-09-26 13:40:31 on dist-test-slave-1wns"
server_key: "2ce194668e74bf50d1550dac3d6c3892"
server_key_iv: "b5c42fbb5fff4fca73c2f3fb3503ee06"
server_key_version: "encryptionkey@0"
I20260926 13:40:31.380143 9388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:40:31.383174 9403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:31.384852 9388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:31.385017 9388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "e4e3ae1ba49d4c53bca332161137d7c4"
format_stamp: "Formatted at 2026-09-26 13:40:31 on dist-test-slave-1wns"
server_key: "2ce194668e74bf50d1550dac3d6c3892"
server_key_iv: "b5c42fbb5fff4fca73c2f3fb3503ee06"
server_key_version: "encryptionkey@0"
I20260926 13:40:31.385154 9388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:31.409919 9388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:31.414256 9388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:31.414499 9388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:31.422276 9388 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:33833
I20260926 13:40:31.422287 9455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:33833 every 8 connection(s)
I20260926 13:40:31.423439 9388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:31.426689 9456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:31.433130 9456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4: Bootstrap starting.
I20260926 13:40:31.433290 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 9388
I20260926 13:40:31.433549 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:31.433852 1220 external_mini_cluster.cc:1428] Setting key 06cbbe4ca45e957afb7f2786174612b8
I20260926 13:40:31.436391 9456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:31.437412 9456 log.cc:824] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:31.439949 9456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4: No bootstrap required, opened a new log
Sep 26 13:40:31 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:31.444782 9456 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33833 } }
I20260926 13:40:31.445125 9456 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:31.445223 9456 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4e3ae1ba49d4c53bca332161137d7c4, State: Initialized, Role: FOLLOWER
I20260926 13:40:31.445843 9456 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33833 } }
I20260926 13:40:31.446054 9456 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:31.446157 9456 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:31.446280 9456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:31.447755 9456 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33833 } }
I20260926 13:40:31.448233 9456 leader_election.cc:302] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4e3ae1ba49d4c53bca332161137d7c4; no voters:
I20260926 13:40:31.448633 9456 leader_election.cc:288] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:31.448794 9461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:31.449026 9461 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 1 LEADER]: Becoming Leader. State: Replica: e4e3ae1ba49d4c53bca332161137d7c4, State: Running, Role: LEADER
I20260926 13:40:31.449404 9461 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33833 } }
I20260926 13:40:31.450275 9456 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:31.451201 9463 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4e3ae1ba49d4c53bca332161137d7c4. Latest consensus state: current_term: 1 leader_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33833 } } }
I20260926 13:40:31.451251 9462 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33833 } } }
I20260926 13:40:31.451473 9462 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:31.451519 9463 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:31.451936 9469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:31.452939 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:31.435349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.1.49.62:33833)
0926 13:40:31.435877 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:31.435900 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:31.435947 (+ 47us) server_negotiation.cc:407] Connection header received
0926 13:40:31.436992 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:31.437018 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:31.437508 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:31.437957 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:31.438877 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:31.440496 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:31.441448 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:31.442277 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:31.442472 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:31.445555 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:31.445563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:31.445583 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:31.445625 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:31.448161 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:31.448842 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:31.448849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:31.448857 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:31.449033 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:31.449379 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:31.449383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:31.449385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:31.450109 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:31.450320 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:31.451118 (+ 798us) server_negotiation.cc:299] Negotiation successful
0926 13:40:31.451494 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":139,"threads_started":1}
I20260926 13:40:31.455363 9469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:31.462108 9469 catalog_manager.cc:1347] Generated new cluster ID: 019769f4688640258c75c57acabcf6fb
I20260926 13:40:31.462216 9469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:31.473639 9469 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:31.474716 9469 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:31.479256 9469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4: Generated new TSK 0
I20260926 13:40:31.480101 9469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:40:31.519569 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33833
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:31.654148 9482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:31.654448 9482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:31.654492 9482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:31.658762 9482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:31.658842 9482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:31.658905 9482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:40:31.664409 9482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:31.665519 9482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:31.666556 9482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:31.677302 9490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:31.677237 9488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:31.677253 9487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:31.677413 9482 server_base.cc:1034] running on GCE node
I20260926 13:40:31.677995 9482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:31.678694 9482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:31.679909 9482 hybrid_clock.cc:648] HybridClock initialized: now 1790430031679880 us; error 34 us; skew 500 ppm
Sep 26 13:40:31 dist-test-slave-1wns krb5kdc[9375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430031, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:31.682963 9482 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:40:31.684007 9482 webserver.cc:460] Webserver started at http://127.1.49.1:37325/ using document root <none> and password file <none>
I20260926 13:40:31.684556 9482 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:31.684616 9482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:31.684798 9482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:31.686930 9482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "5679043ca2f74ea5b3b2ca1a4424a2d9"
format_stamp: "Formatted at 2026-09-26 13:40:31 on dist-test-slave-1wns"
server_key: "fa3c5e8f877cd8465fd73c736cf25087"
server_key_iv: "879de48bd6ab4b64df16f44482f873a0"
server_key_version: "encryptionkey@0"
I20260926 13:40:31.687539 9482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "5679043ca2f74ea5b3b2ca1a4424a2d9"
format_stamp: "Formatted at 2026-09-26 13:40:31 on dist-test-slave-1wns"
server_key: "fa3c5e8f877cd8465fd73c736cf25087"
server_key_iv: "879de48bd6ab4b64df16f44482f873a0"
server_key_version: "encryptionkey@0"
I20260926 13:40:31.692027 9482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 13:40:31.694711 9497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:31.696447 9482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:31.696573 9482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "5679043ca2f74ea5b3b2ca1a4424a2d9"
format_stamp: "Formatted at 2026-09-26 13:40:31 on dist-test-slave-1wns"
server_key: "fa3c5e8f877cd8465fd73c736cf25087"
server_key_iv: "879de48bd6ab4b64df16f44482f873a0"
server_key_version: "encryptionkey@0"
I20260926 13:40:31.696681 9482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:31.707564 9482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:31.711450 9482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:31.711630 9482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:31.712314 9482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:31.713333 9482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:31.713416 9482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:31.713497 9482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:31.713541 9482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:31.725148 9482 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:42277
I20260926 13:40:31.725162 9610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:42277 every 8 connection(s)
I20260926 13:40:31.726403 9482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:40:31.728025 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 9482
I20260926 13:40:31.728188 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:40:31.728639 1220 external_mini_cluster.cc:1428] Setting key d01674a5ad56f26c75fd165946d87aad
Sep 26 13:40:31 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430031, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
I20260926 13:40:31.744875 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:31.728665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:56657 (local address 127.1.49.62:33833)
0926 13:40:31.728845 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:31.728851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:31.729841 (+ 990us) server_negotiation.cc:407] Connection header received
0926 13:40:31.732029 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:31.732032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:31.732083 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:31.732159 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:31.733653 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:31.734197 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:31.736403 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:31.736641 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:31.736739 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:31.740302 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:31.740307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:31.740310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:31.740352 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:31.742067 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:31.742953 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:31.742957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:31.742959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:31.743053 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:31.743601 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:31.743605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:31.743607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:31.743873 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:31.743985 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:31.744556 (+ 571us) server_negotiation.cc:299] Negotiation successful
0926 13:40:31.744702 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 13:40:31.745819 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:31.729098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:56657)
0926 13:40:31.729622 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:31.729690 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:31.731696 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:31.732395 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:31.732409 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:31.732919 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:31.733494 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:31.733511 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:31.734381 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:31.734389 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:31.736230 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:31.736246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:31.736885 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:31.736894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:31.737166 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:31.737970 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:31.738000 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:31.740026 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:31.742251 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:31.742260 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:31.742282 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:31.742799 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:31.743235 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:31.743243 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:31.743244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:31.743442 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:31.743995 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:31.744004 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:31.744289 (+ 285us) client_negotiation.cc:769] Sending connection context
0926 13:40:31.744494 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 13:40:31.744863 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":118,"threads_started":1}
I20260926 13:40:31.747407 9611 heartbeater.cc:347] Connected to a master server at 127.1.49.62:33833
I20260926 13:40:31.747730 9611 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:31.748492 9611 heartbeater.cc:511] Master 127.1.49.62:33833 requested a full tablet report, sending...
I20260926 13:40:31.750500 9420 ts_manager.cc:194] Registered new tserver with Master: 5679043ca2f74ea5b3b2ca1a4424a2d9 (127.1.49.1:42277)
I20260926 13:40:31.751950 9420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:56657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:40:31.764008 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33833
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:31.885192 9616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:31.885672 9616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:31.885775 9616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:31.892042 9616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:31.892133 9616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:31.892238 9616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:40:31.897886 9616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:31.899072 9616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:31.900271 9616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:31.910459 9622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:31.910544 9616 server_base.cc:1034] running on GCE node
W20260926 13:40:31.910459 9621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:31.910614 9624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:31.911286 9616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:31.912084 9616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:31.913278 9616 hybrid_clock.cc:648] HybridClock initialized: now 1790430031913256 us; error 37 us; skew 500 ppm
Sep 26 13:40:31 dist-test-slave-1wns krb5kdc[9375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430031, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:31.916770 9616 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:40:31.917900 9616 webserver.cc:460] Webserver started at http://127.1.49.2:33993/ using document root <none> and password file <none>
I20260926 13:40:31.918624 9616 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:31.918697 9616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:31.918996 9616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:31.921445 9616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "17e57b4f84664a6b9f0a904181a60180"
format_stamp: "Formatted at 2026-09-26 13:40:31 on dist-test-slave-1wns"
server_key: "fe404e7eed86bc08925c8855e007c1a2"
server_key_iv: "c7aa8ea9549c340c2e0c1cc11d0d0ffc"
server_key_version: "encryptionkey@0"
I20260926 13:40:31.922112 9616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "17e57b4f84664a6b9f0a904181a60180"
format_stamp: "Formatted at 2026-09-26 13:40:31 on dist-test-slave-1wns"
server_key: "fe404e7eed86bc08925c8855e007c1a2"
server_key_iv: "c7aa8ea9549c340c2e0c1cc11d0d0ffc"
server_key_version: "encryptionkey@0"
I20260926 13:40:31.926772 9616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:40:31.930356 9631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:31.932475 9616 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:40:31.932646 9616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "17e57b4f84664a6b9f0a904181a60180"
format_stamp: "Formatted at 2026-09-26 13:40:31 on dist-test-slave-1wns"
server_key: "fe404e7eed86bc08925c8855e007c1a2"
server_key_iv: "c7aa8ea9549c340c2e0c1cc11d0d0ffc"
server_key_version: "encryptionkey@0"
I20260926 13:40:31.932801 9616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:31.947633 9616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:31.951529 9616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:31.951763 9616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:31.952474 9616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:31.953418 9616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:31.953490 9616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:31.953558 9616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:31.953603 9616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:31.963204 9616 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:36627
I20260926 13:40:31.963234 9744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:36627 every 8 connection(s)
I20260926 13:40:31.964216 9616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:40:31.971940 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 9616
I20260926 13:40:31.972095 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:40:31.972483 1220 external_mini_cluster.cc:1428] Setting key d46a6454c7ac9622b876a27fca2deb88
Sep 26 13:40:31 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430031, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:31.980973 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:31.966423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:36849 (local address 127.1.49.62:33833)
0926 13:40:31.966597 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:31.966602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:31.967580 (+ 978us) server_negotiation.cc:407] Connection header received
0926 13:40:31.969265 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:31.969270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:31.969364 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:31.969477 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:31.970912 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:31.971519 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:31.973254 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:31.973515 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:31.973656 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:31.976705 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:31.976712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:31.976718 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:31.976760 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:31.978474 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:31.979127 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:31.979131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:31.979132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:31.979222 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:31.979654 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:31.979658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:31.979660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:31.979947 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:31.980052 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:31.980616 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 13:40:31.980782 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 13:40:31.981871 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:31.966825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:36849)
0926 13:40:31.967389 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:31.967441 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:31.968977 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:31.969685 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:31.969693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:31.970130 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:31.970695 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:31.970710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:31.971712 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:31.971718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:31.973101 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:31.973120 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:31.973712 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:31.973717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:31.973918 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:31.974658 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:31.974683 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:31.976488 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:31.978626 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:31.978633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:31.978653 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:31.979002 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:31.979324 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:31.979328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:31.979330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:31.979547 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:31.980062 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:31.980071 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:31.980365 (+ 294us) client_negotiation.cc:769] Sending connection context
0926 13:40:31.980564 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 13:40:31.980948 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
I20260926 13:40:31.983388 9745 heartbeater.cc:347] Connected to a master server at 127.1.49.62:33833
I20260926 13:40:31.983666 9745 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:31.984299 9745 heartbeater.cc:511] Master 127.1.49.62:33833 requested a full tablet report, sending...
I20260926 13:40:31.985478 9420 ts_manager.cc:194] Registered new tserver with Master: 17e57b4f84664a6b9f0a904181a60180 (127.1.49.2:36627)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
I20260926 13:40:31.986171 9420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:36849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:40:32.003870 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33833
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:32.122637 9750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:32.122970 9750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:32.123020 9750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:32.127322 9750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:32.127415 9750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:32.127480 9750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:40:32.133304 9750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:32.134987 9750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:32.136157 9750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:32.147336 9756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:32.147333 9755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:32.147333 9758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:32.147663 9750 server_base.cc:1034] running on GCE node
I20260926 13:40:32.148283 9750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:32.149017 9750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:32.150269 9750 hybrid_clock.cc:648] HybridClock initialized: now 1790430032150238 us; error 42 us; skew 500 ppm
Sep 26 13:40:32 dist-test-slave-1wns krb5kdc[9375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430032, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:32.153570 9750 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:40:32.154527 9750 webserver.cc:460] Webserver started at http://127.1.49.3:36533/ using document root <none> and password file <none>
I20260926 13:40:32.155112 9750 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:32.155172 9750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:32.155434 9750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:32.157687 9750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "4dd971b575db4c72af1c3c9d9b14fd83"
format_stamp: "Formatted at 2026-09-26 13:40:32 on dist-test-slave-1wns"
server_key: "d039b57498b5b7af80253df921f907aa"
server_key_iv: "c250956610063ab423d557d2351f047a"
server_key_version: "encryptionkey@0"
I20260926 13:40:32.158277 9750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "4dd971b575db4c72af1c3c9d9b14fd83"
format_stamp: "Formatted at 2026-09-26 13:40:32 on dist-test-slave-1wns"
server_key: "d039b57498b5b7af80253df921f907aa"
server_key_iv: "c250956610063ab423d557d2351f047a"
server_key_version: "encryptionkey@0"
I20260926 13:40:32.162235 9750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 13:40:32.165093 9765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:32.166618 9750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:32.166841 9750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "4dd971b575db4c72af1c3c9d9b14fd83"
format_stamp: "Formatted at 2026-09-26 13:40:32 on dist-test-slave-1wns"
server_key: "d039b57498b5b7af80253df921f907aa"
server_key_iv: "c250956610063ab423d557d2351f047a"
server_key_version: "encryptionkey@0"
I20260926 13:40:32.167001 9750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:32.185456 9750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:32.189050 9750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:32.189298 9750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:32.190001 9750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:32.190936 9750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:32.191007 9750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:32.191074 9750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:32.191105 9750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:32.200994 9750 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:40595
I20260926 13:40:32.201014 9878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:40595 every 8 connection(s)
I20260926 13:40:32.202039 9750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:40:32.211609 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 9750
I20260926 13:40:32.211788 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:40:32.212158 1220 external_mini_cluster.cc:1428] Setting key fa139f5eb29f9d85aa0f17d30bd32d80
Sep 26 13:40:32 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430032, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:32.218062 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:32.204456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:47681 (local address 127.1.49.62:33833)
0926 13:40:32.204632 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:32.204636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:32.205575 (+ 939us) server_negotiation.cc:407] Connection header received
0926 13:40:32.206990 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:32.206993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:32.207052 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:32.207164 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:32.208457 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.209003 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.210186 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.210410 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.210541 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:32.213895 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:32.213899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:32.213901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:32.213927 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:32.215707 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:32.216331 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:32.216336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:32.216338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:32.216430 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:32.216818 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:32.216822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:32.216824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:32.217031 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:32.217149 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:32.217731 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 13:40:32.217876 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 13:40:32.218968 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:32.204745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:47681)
0926 13:40:32.205311 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:32.205358 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:32.206746 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:32.207355 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:32.207362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:32.207740 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:32.208288 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:32.208306 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.209136 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.209142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:32.210046 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:32.210055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.210573 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.210577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:32.210784 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:32.211546 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:32.211565 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:32.213688 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:32.215863 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:32.215871 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:32.215890 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:32.216211 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:32.216533 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:32.216537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:32.216539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:32.216704 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:32.217147 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:32.217156 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:32.217454 (+ 298us) client_negotiation.cc:769] Sending connection context
0926 13:40:32.217662 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 13:40:32.218036 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":136,"threads_started":1}
I20260926 13:40:32.220376 9879 heartbeater.cc:347] Connected to a master server at 127.1.49.62:33833
I20260926 13:40:32.220666 9879 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:32.221223 9879 heartbeater.cc:511] Master 127.1.49.62:33833 requested a full tablet report, sending...
I20260926 13:40:32.222468 9420 ts_manager.cc:194] Registered new tserver with Master: 4dd971b575db4c72af1c3c9d9b14fd83 (127.1.49.3:40595)
I20260926 13:40:32.223075 9420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:47681
I20260926 13:40:32.226964 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.1.keytab.
Entry for principal oryx/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.1.keytab.
WARNING: no policy specified for oryx/127.1.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.2.keytab.
Entry for principal oryx/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.2.keytab.
WARNING: no policy specified for oryx/127.1.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.49.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.3.keytab.
Entry for principal oryx/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.3.keytab.
I20260926 13:40:32.310845 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 9482
I20260926 13:40:32.317039 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 9616
I20260926 13:40:32.322974 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 9750
I20260926 13:40:32.328858 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 9388
I20260926 13:40:32.335582 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:33833
--webserver_interface=127.1.49.62
--webserver_port=40933
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:33833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:32.451638 9888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:32.452035 9888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:32.452087 9888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:32.457386 9888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:32.457479 9888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:32.457516 9888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:32.457559 9888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:32.457840 9888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:32.469846 9888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:33833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:33833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=40933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:32.471679 9888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:32.473289 9888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:32.484807 9894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:32.484874 9893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:32.485157 9896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:32.485410 9888 server_base.cc:1034] running on GCE node
I20260926 13:40:32.486058 9888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:32.487639 9888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:32.488806 9888 hybrid_clock.cc:648] HybridClock initialized: now 1790430032488789 us; error 54 us; skew 500 ppm
Sep 26 13:40:32 dist-test-slave-1wns krb5kdc[9375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430032, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:32.491739 9888 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:40:32.493135 9888 webserver.cc:460] Webserver started at http://127.1.49.62:40933/ using document root <none> and password file <none>
I20260926 13:40:32.493965 9888 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:32.494071 9888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:32.499891 9888 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:40:32.502403 9903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:32.503916 9888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:32.504050 9888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "e4e3ae1ba49d4c53bca332161137d7c4"
format_stamp: "Formatted at 2026-09-26 13:40:31 on dist-test-slave-1wns"
server_key: "2ce194668e74bf50d1550dac3d6c3892"
server_key_iv: "b5c42fbb5fff4fca73c2f3fb3503ee06"
server_key_version: "encryptionkey@0"
I20260926 13:40:32.504640 9888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:32.548588 9888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:32.552250 9888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:32.552484 9888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:32.560062 9888 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:33833
I20260926 13:40:32.560079 9955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:33833 every 8 connection(s)
I20260926 13:40:32.561229 9888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:32.563381 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 9888
I20260926 13:40:32.563876 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:42277
--local_ip_for_outbound_sockets=127.1.49.1
--tserver_master_addrs=127.1.49.62:33833
--webserver_port=37325
--webserver_interface=127.1.49.1
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.49.1
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:40:32.566195 9956 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:40:32.568208 9956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4: Bootstrap starting.
I20260926 13:40:32.587006 9956 log.cc:824] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:32.594565 9956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:40:32.595093 9956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4: Bootstrap complete.
I20260926 13:40:32.599885 9956 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33833 } }
I20260926 13:40:32.600368 9956 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4e3ae1ba49d4c53bca332161137d7c4, State: Initialized, Role: FOLLOWER
I20260926 13:40:32.600955 9956 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33833 } }
I20260926 13:40:32.601078 9956 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:32.601190 9956 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:32.601290 9956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:40:32.603493 9956 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33833 } }
I20260926 13:40:32.603952 9956 leader_election.cc:302] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4e3ae1ba49d4c53bca332161137d7c4; no voters:
I20260926 13:40:32.604349 9956 leader_election.cc:288] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:40:32.604450 9960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:40:32.604643 9960 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [term 2 LEADER]: Becoming Leader. State: Replica: e4e3ae1ba49d4c53bca332161137d7c4, State: Running, Role: LEADER
I20260926 13:40:32.605012 9960 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33833 } }
I20260926 13:40:32.605772 9956 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:32.606097 9960 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4e3ae1ba49d4c53bca332161137d7c4. Latest consensus state: current_term: 2 leader_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33833 } } }
I20260926 13:40:32.606269 9960 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:32.606635 9961 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3ae1ba49d4c53bca332161137d7c4" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33833 } } }
I20260926 13:40:32.606797 9961 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:32.607122 9964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:32.610945 9964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:32.612108 9964 catalog_manager.cc:1259] Loaded cluster ID: 019769f4688640258c75c57acabcf6fb
I20260926 13:40:32.612181 9964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:32.614511 9964 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:32.615197 9964 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e4e3ae1ba49d4c53bca332161137d7c4: Loaded TSK: 0
I20260926 13:40:32.616613 9964 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:40:32.690042 9958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:32.690337 9958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:32.690385 9958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:32.694857 9958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:32.694953 9958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:32.695019 9958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:40:32.701018 9958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.1.keytab
--principal=oryx/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:42277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=37325
--tserver_master_addrs=127.1.49.62:33833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:32.702200 9958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:32.703320 9958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:32.714401 9982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:32.714349 9985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:32.714330 9983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:32.714617 9958 server_base.cc:1034] running on GCE node
I20260926 13:40:32.715155 9958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:32.716010 9958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:32.717204 9958 hybrid_clock.cc:648] HybridClock initialized: now 1790430032717160 us; error 66 us; skew 500 ppm
Sep 26 13:40:32 dist-test-slave-1wns krb5kdc[9375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430032, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:32.720667 9958 init.cc:377] Logged in from keytab as oryx/127.1.49.1@KRBTEST.COM (short username oryx)
I20260926 13:40:32.722205 9958 webserver.cc:460] Webserver started at http://127.1.49.1:37325/ using document root <none> and password file <none>
I20260926 13:40:32.723029 9958 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:32.723147 9958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:32.729104 9958 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:40:32.732442 9992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:32.734339 9958 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:40:32.734514 9958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "5679043ca2f74ea5b3b2ca1a4424a2d9"
format_stamp: "Formatted at 2026-09-26 13:40:31 on dist-test-slave-1wns"
server_key: "fa3c5e8f877cd8465fd73c736cf25087"
server_key_iv: "879de48bd6ab4b64df16f44482f873a0"
server_key_version: "encryptionkey@0"
I20260926 13:40:32.735371 9958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:32.752238 9958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:32.756513 9958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:32.756745 9958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:32.757524 9958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:32.758665 9958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:32.758777 9958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:32.758841 9958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:32.758865 9958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:32.771484 9958 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:42277
I20260926 13:40:32.771489 10105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:42277 every 8 connection(s)
I20260926 13:40:32.773240 9958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:40:32.781607 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 9958
I20260926 13:40:32.782159 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:36627
--local_ip_for_outbound_sockets=127.1.49.2
--tserver_master_addrs=127.1.49.62:33833
--webserver_port=33993
--webserver_interface=127.1.49.2
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.49.2
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:40:32 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:32 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:40:32.791467 10108 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:32.775961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:56027 (local address 127.1.49.62:33833)
0926 13:40:32.776659 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:32.776682 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:32.777275 (+ 593us) server_negotiation.cc:407] Connection header received
0926 13:40:32.778676 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:32.778689 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:32.779032 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:32.779337 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:32.781052 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.782568 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.784613 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.785391 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.785588 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:32.790679 (+ 5091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:56027: BlockingRecv error: recv got EOF from 127.1.49.1:56027 (error 108)
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":136,"threads_started":1}
I20260926 13:40:32.791643 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:32.776058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:56027)
0926 13:40:32.777053 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:32.777113 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:32.778236 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:32.779520 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:32.779530 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:32.780136 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:32.780824 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:32.780844 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.782768 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.782774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:32.784429 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:32.784445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.785609 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.785616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:32.785888 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:32.786724 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:32.786755 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:32.790417 (+ 3662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":333,"threads_started":1}
W20260926 13:40:32.792294 10106 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:33833 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.49.62:33833: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Sep 26 13:40:32 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:32 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:32.798568 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:32.792814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:55215)
0926 13:40:32.793107 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:32.793124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:32.793235 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:32.793572 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:32.793575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:32.793766 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:32.793879 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:32.793884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.794660 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.794663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:32.795243 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:32.795251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.795723 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.795726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:32.795852 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:32.796401 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:32.796416 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:32.798401 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227}
W20260926 13:40:32.798808 10108 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:32.792960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:55215 (local address 127.1.49.62:33833)
0926 13:40:32.793091 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:32.793096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:32.793180 (+ 84us) server_negotiation.cc:407] Connection header received
0926 13:40:32.793404 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:32.793407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:32.793456 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:32.793557 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:32.794009 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.794544 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.795368 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.795597 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.795718 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:32.798558 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:55215: BlockingRecv error: recv got EOF from 127.1.49.1:55215 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 13:40:32 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:32 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:32.804157 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:32.799099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:43263)
0926 13:40:32.799233 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:32.799250 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:32.799330 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:32.799564 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:32.799567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:32.799724 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:32.799852 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:32.799861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.800591 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.800596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:32.801139 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:32.801147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.801599 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.801603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:32.801684 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:32.802199 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:32.802219 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:32.804026 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 13:40:32.804546 10108 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:32.799227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:43263 (local address 127.1.49.62:33833)
0926 13:40:32.799350 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:32.799356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:32.799373 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:40:32.799413 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:32.799417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:32.799461 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:32.799546 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:32.799974 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.800483 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.801269 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.801484 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.801593 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:32.804395 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:43263: BlockingRecv error: recv got EOF from 127.1.49.1:43263 (error 108)
Metrics: {"server-negotiator.queue_time_us":60,"thread_start_us":131,"threads_started":1}
W20260926 13:40:32.804886 10106 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:40:32.906198 10110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:32.906544 10110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:32.906596 10110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:32.911399 10110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:32.911474 10110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:32.911537 10110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:40:32.917127 10110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.2.keytab
--principal=oryx/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:36627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=33993
--tserver_master_addrs=127.1.49.62:33833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:32.918304 10110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:32.919390 10110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:32.930415 10116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:32.930435 10119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:32.930435 10117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:32.930498 10110 server_base.cc:1034] running on GCE node
I20260926 13:40:32.931142 10110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:32.931885 10110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:32.933076 10110 hybrid_clock.cc:648] HybridClock initialized: now 1790430032933054 us; error 36 us; skew 500 ppm
Sep 26 13:40:32 dist-test-slave-1wns krb5kdc[9375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430032, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:32.937453 10110 init.cc:377] Logged in from keytab as oryx/127.1.49.2@KRBTEST.COM (short username oryx)
I20260926 13:40:32.938604 10110 webserver.cc:460] Webserver started at http://127.1.49.2:33993/ using document root <none> and password file <none>
I20260926 13:40:32.939208 10110 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:32.939301 10110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:32.943969 10110 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:40:32.946482 10126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:32.948271 10110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:32.948453 10110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "17e57b4f84664a6b9f0a904181a60180"
format_stamp: "Formatted at 2026-09-26 13:40:31 on dist-test-slave-1wns"
server_key: "fe404e7eed86bc08925c8855e007c1a2"
server_key_iv: "c7aa8ea9549c340c2e0c1cc11d0d0ffc"
server_key_version: "encryptionkey@0"
I20260926 13:40:32.949091 10110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:32.971187 10110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:32.975031 10110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:32.975270 10110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:32.975966 10110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:32.976969 10110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:32.977036 10110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:32.977108 10110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:32.977159 10110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:32.987171 10110 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:36627
I20260926 13:40:32.987187 10239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:36627 every 8 connection(s)
I20260926 13:40:32.988385 10110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:40:32.991225 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 10110
I20260926 13:40:32.991727 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:40595
--local_ip_for_outbound_sockets=127.1.49.3
--tserver_master_addrs=127.1.49.62:33833
--webserver_port=36533
--webserver_interface=127.1.49.3
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.49.3
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:40:32 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:32 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:40:33.000936 10108 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:32.990281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:45437 (local address 127.1.49.62:33833)
0926 13:40:32.990436 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:32.990441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:32.991681 (+ 1240us) server_negotiation.cc:407] Connection header received
0926 13:40:32.993185 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:32.993189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:32.993256 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:32.993338 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:32.994658 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.995202 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.996524 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.996758 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.996898 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:33.000790 (+ 3892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:45437: BlockingRecv error: recv got EOF from 127.1.49.2:45437 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:40:33.001773 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:32.990819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:45437)
0926 13:40:32.991500 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:32.991549 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:32.992929 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:32.993546 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:32.993554 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:32.993956 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:32.994488 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:32.994503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.995360 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.995363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:32.996384 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:32.996394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:32.996950 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:32.996954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:32.997147 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:32.997860 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:32.997877 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:33.000593 (+ 2716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":150,"threads_started":1}
W20260926 13:40:33.002413 10240 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:33833 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.49.62:33833: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Sep 26 13:40:33 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:33 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:33.010782 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:33.002970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:59811)
0926 13:40:33.003240 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:33.003260 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:33.003365 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:33.003677 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:33.003681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:33.003932 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:33.004077 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:33.004085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.004978 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.004982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:33.006068 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:33.006082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.006813 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.006818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:33.006955 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:33.007658 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:33.007678 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:33.010547 (+ 2869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180}
W20260926 13:40:33.011045 10108 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:33.003117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:59811 (local address 127.1.49.62:33833)
0926 13:40:33.003283 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:33.003287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:33.003319 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:40:33.003481 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:33.003483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:33.003531 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:33.003617 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:33.004249 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.004844 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.006387 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.006676 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.006820 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:33.010879 (+ 4059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:59811: BlockingRecv error: recv got EOF from 127.1.49.2:59811 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 13:40:33 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:33 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:33.018174 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:33.011592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:41457)
0926 13:40:33.011956 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:33.011973 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:33.012077 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:33.012320 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:33.012323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:33.012511 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:33.012621 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:33.012628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.013485 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.013489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:33.014394 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:33.014405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.014921 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.014926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:33.015022 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:33.015570 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:33.015587 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:33.018029 (+ 2442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277}
W20260926 13:40:33.018410 10240 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:40:33.018576 10108 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:33.011744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:41457 (local address 127.1.49.62:33833)
0926 13:40:33.011908 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:33.011912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:33.012025 (+ 113us) server_negotiation.cc:407] Connection header received
0926 13:40:33.012166 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:33.012168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:33.012207 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:33.012279 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:33.012779 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.013364 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.014566 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.014788 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.014895 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:33.018183 (+ 3288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:41457: BlockingRecv error: recv got EOF from 127.1.49.2:41457 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 13:40:33.122385 10243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:33.122720 10243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:33.122807 10243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:33.127053 10243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:33.127162 10243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:33.127264 10243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:40:33.133389 10243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/krb5kdc/oryx_127.1.49.3.keytab
--principal=oryx/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:40595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=36533
--tserver_master_addrs=127.1.49.62:33833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:33.134568 10243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:33.135658 10243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:33.146744 10251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:33.146726 10249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:33.146816 10243 server_base.cc:1034] running on GCE node
W20260926 13:40:33.146724 10248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:33.147475 10243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:33.148263 10243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:33.149452 10243 hybrid_clock.cc:648] HybridClock initialized: now 1790430033149415 us; error 52 us; skew 500 ppm
Sep 26 13:40:33 dist-test-slave-1wns krb5kdc[9375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430033, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:33.152630 10243 init.cc:377] Logged in from keytab as oryx/127.1.49.3@KRBTEST.COM (short username oryx)
I20260926 13:40:33.153687 10243 webserver.cc:460] Webserver started at http://127.1.49.3:36533/ using document root <none> and password file <none>
I20260926 13:40:33.154377 10243 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:33.154484 10243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:33.158577 10243 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:40:33.161154 10258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:33.162581 10243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:33.162789 10243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "4dd971b575db4c72af1c3c9d9b14fd83"
format_stamp: "Formatted at 2026-09-26 13:40:32 on dist-test-slave-1wns"
server_key: "d039b57498b5b7af80253df921f907aa"
server_key_iv: "c250956610063ab423d557d2351f047a"
server_key_version: "encryptionkey@0"
I20260926 13:40:33.163429 10243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:33.189363 10243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:33.193678 10243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:33.193933 10243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:33.194639 10243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:33.195622 10243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:33.195700 10243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:33.195781 10243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:33.195873 10243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 13:40:33.206902 10243 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:40595
I20260926 13:40:33.206970 10371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:40595 every 8 connection(s)
I20260926 13:40:33.208340 10243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:40:33.209892 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 10243
Sep 26 13:40:33 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:33 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:33.221498 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:33.211048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.1.49.62:33833)
0926 13:40:33.211171 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:33.211190 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:33.211275 (+ 85us) server_negotiation.cc:407] Connection header received
0926 13:40:33.211321 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:33.211324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:33.211368 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:33.211511 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:33.212454 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.213375 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.214417 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.214744 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.214869 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:33.216574 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:33.216582 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:33.216601 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:33.216638 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:33.218997 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:33.219468 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:33.219475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:33.219483 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:33.219668 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:33.220074 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:33.220080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:33.220082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:33.220517 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:33.220749 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:33.221082 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 13:40:33.221361 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 13:40:33.221989 10108 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:33.210593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:49237 (local address 127.1.49.62:33833)
0926 13:40:33.210789 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:33.210794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:33.212200 (+ 1406us) server_negotiation.cc:407] Connection header received
0926 13:40:33.213519 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:33.213524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:33.213580 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:33.213679 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:33.215172 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.215983 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.217831 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.218050 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.218132 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:33.221890 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:49237: BlockingRecv error: recv got EOF from 127.1.49.3:49237 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 13:40:33.222671 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:33.211139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:49237)
0926 13:40:33.212041 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:33.212090 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:33.213212 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:33.213899 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:33.213907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:33.214296 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:33.215009 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:33.215033 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.216131 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.216134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:33.217664 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:33.217675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.218214 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.218219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:33.218436 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:33.218997 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:33.219019 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:33.221761 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":116,"threads_started":1}
W20260926 13:40:33.223284 10372 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:33833 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.49.62:33833: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Sep 26 13:40:33 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:33 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:33.229529 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:33.223760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:44099)
0926 13:40:33.223915 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:33.223932 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:33.224023 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:33.224402 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:33.224406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:33.224594 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:33.224727 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:33.224748 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.225587 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.225590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:33.226209 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:33.226216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.226722 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.226726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:33.226830 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:33.227333 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:33.227346 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:33.229356 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 13:40:33.229621 10111 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:33.224030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:44099 (local address 127.1.49.62:33833)
0926 13:40:33.224141 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:33.224145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:33.224158 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:40:33.224197 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:33.224202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:33.224281 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:33.224363 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:33.224886 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.225461 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.226335 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.226605 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.226723 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:33.229497 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:44099: BlockingRecv error: recv got EOF from 127.1.49.3:44099 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 13:40:33 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:33 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:33.235513 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:33.230099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:55643)
0926 13:40:33.230218 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:33.230234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:33.230310 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:33.230581 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:33.230584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:33.230771 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:33.230865 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:33.230870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.231727 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.231730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:33.232384 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:33.232397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.232819 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.232822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:33.232886 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:33.233523 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:33.233535 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:33.235407 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 13:40:33.235775 10111 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:33.230200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:55643 (local address 127.1.49.62:33833)
0926 13:40:33.230347 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:33.230352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:33.230370 (+ 18us) server_negotiation.cc:407] Connection header received
0926 13:40:33.230405 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:33.230407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:33.230445 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:33.230516 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:33.230996 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.231569 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.232493 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.232698 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.232782 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:33.235627 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:55643: BlockingRecv error: recv got EOF from 127.1.49.3:55643 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 13:40:33.235903 10372 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 13:40:33 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:33 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:33.812151 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:33.805706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:40751)
0926 13:40:33.806014 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:33.806030 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:33.806137 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:33.806452 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:33.806454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:33.806679 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:33.806836 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:33.806842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.807724 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.807727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:33.808489 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:33.808500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.809001 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.809004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:33.809076 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:33.809753 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:33.809766 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:33.811957 (+ 2191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":99,"threads_started":1}
W20260926 13:40:33.812561 10377 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:33.805810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:40751 (local address 127.1.49.62:33833)
0926 13:40:33.806110 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:33.806115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:33.806132 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:40:33.806231 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:33.806234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:33.806284 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:33.806417 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:33.806994 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.807621 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.808586 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:33.808885 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:33.808974 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:33.812408 (+ 3434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:40751: BlockingRecv error: recv got EOF from 127.1.49.1:40751 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":111,"threads_started":1}
Sep 26 13:40:34 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:34 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:34.026137 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:34.019325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:45021)
0926 13:40:34.019642 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:34.019662 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:34.019927 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:34.020270 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:34.020273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:34.020507 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:34.020605 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:34.020611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:34.021539 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:34.021547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:34.022350 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:34.022363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:34.023020 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:34.023026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:34.023115 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:34.023776 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:34.023792 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:34.025993 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":93,"threads_started":1}
W20260926 13:40:34.026346 10377 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:34.019396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:45021 (local address 127.1.49.62:33833)
0926 13:40:34.019618 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:34.019623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:34.019764 (+ 141us) server_negotiation.cc:407] Connection header received
0926 13:40:34.020057 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:34.020060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:34.020120 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:34.020246 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:34.020739 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:34.021408 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:34.022572 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:34.022879 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:34.022982 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:34.026187 (+ 3205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:45021: BlockingRecv error: recv got EOF from 127.1.49.2:45021 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 26 13:40:34 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:34 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:34.243018 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:34.236648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:53895)
0926 13:40:34.236943 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:34.236961 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:34.237076 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:34.237502 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:34.237505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:34.237724 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:34.237840 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:34.237847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:34.238821 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:34.238824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:34.239482 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:34.239490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:34.240061 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:34.240065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:34.240166 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:34.240724 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:34.240735 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:34.242810 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":144,"threads_started":1}
W20260926 13:40:34.243093 10377 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:34.236741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:53895 (local address 127.1.49.62:33833)
0926 13:40:34.236935 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:34.236939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:34.237076 (+ 137us) server_negotiation.cc:407] Connection header received
0926 13:40:34.237155 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:34.237157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:34.237276 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:34.237443 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:34.237984 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:34.238661 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:34.239607 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:34.239875 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:34.240021 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:34.242971 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:53895: BlockingRecv error: recv got EOF from 127.1.49.3:53895 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 26 13:40:34 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:34 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:34.820114 10380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:34.813603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:36031)
0926 13:40:34.813960 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:34.813997 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:34.814117 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:34.814486 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:34.814490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:34.814722 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:34.814859 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:34.814866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:34.815733 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:34.815737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:34.816388 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:34.816399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:34.817083 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:34.817086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:34.817159 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:34.817812 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:34.817825 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:34.819972 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":135,"threads_started":1}
W20260926 13:40:34.820307 10381 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:34.813707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:36031 (local address 127.1.49.62:33833)
0926 13:40:34.814069 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:34.814072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:34.814085 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:40:34.814232 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:34.814235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:34.814289 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:34.814393 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:34.814985 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:34.815598 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:34.816680 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:34.816933 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:34.817032 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:34.820144 (+ 3112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:36031: BlockingRecv error: recv got EOF from 127.1.49.1:36031 (error 108)
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
Sep 26 13:40:35 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:35 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:35.033195 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:35.027060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:41833)
0926 13:40:35.027432 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:35.027446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:35.027548 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:35.027875 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:35.027877 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:35.028056 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:35.028152 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:35.028157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:35.028997 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:35.029000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:35.029625 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:35.029634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:35.030121 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:35.030124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:35.030190 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:35.030831 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:35.030843 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:35.033042 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":100,"threads_started":1}
W20260926 13:40:35.033347 10381 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:35.027149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:41833 (local address 127.1.49.62:33833)
0926 13:40:35.027422 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:35.027426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:35.027533 (+ 107us) server_negotiation.cc:407] Connection header received
0926 13:40:35.027648 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:35.027659 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:35.027713 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:35.027821 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:35.028304 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:35.028877 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:35.029751 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:35.030004 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:35.030093 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:35.033218 (+ 3125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:41833: BlockingRecv error: recv got EOF from 127.1.49.2:41833 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 26 13:40:35 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:35 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:35.250849 10383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:35.244006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:34601)
0926 13:40:35.244521 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:35.244537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:35.244693 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:35.245071 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:35.245074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:35.245324 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:35.245425 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:35.245431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:35.246311 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:35.246318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:35.246975 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:35.246985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:35.247519 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:35.247523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:35.247619 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:35.248375 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:35.248392 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:35.250670 (+ 2278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"spinlock_wait_cycles":768,"thread_start_us":195,"threads_started":1}
W20260926 13:40:35.251120 10381 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:35.244233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:34601 (local address 127.1.49.62:33833)
0926 13:40:35.244456 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:35.244462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:35.244624 (+ 162us) server_negotiation.cc:407] Connection header received
0926 13:40:35.244863 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:35.244871 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:35.244932 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:35.245039 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:35.245572 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:35.246160 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:35.247134 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:35.247390 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:35.247516 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:35.250982 (+ 3466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:34601: BlockingRecv error: recv got EOF from 127.1.49.3:34601 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 13:40:35 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:35 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:40:35.828298 10385 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:35.821254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:43065 (local address 127.1.49.62:33833)
0926 13:40:35.821573 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:35.821577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:35.821591 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:40:35.821635 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:35.821638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:35.821682 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:35.821785 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:35.822651 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:35.823271 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:35.824363 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:35.824593 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:35.824681 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:35.828169 (+ 3488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:43065: BlockingRecv error: recv got EOF from 127.1.49.1:43065 (error 108)
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":97,"threads_started":1}
I20260926 13:40:35.828213 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:35.821087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:43065)
0926 13:40:35.821392 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:35.821411 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:35.821523 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:35.821852 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:35.821854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:35.822346 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:35.822454 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:35.822461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:35.823424 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:35.823427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:35.824143 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:35.824229 (+ 86us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:35.824748 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:35.824751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:35.824823 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:35.825507 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:35.825519 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:35.828009 (+ 2490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":131,"threads_started":1}
Sep 26 13:40:36 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:36 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:36.041222 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:36.034445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:56153)
0926 13:40:36.034778 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:36.034793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:36.034910 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:36.035234 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:36.035236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:36.035478 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:36.035575 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:36.035581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:36.036631 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:36.036633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:36.037227 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:36.037237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:36.037805 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:36.037808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:36.037905 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:36.038567 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:36.038580 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:36.041021 (+ 2441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":125,"threads_started":1}
W20260926 13:40:36.041373 10385 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:36.034539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:56153 (local address 127.1.49.62:33833)
0926 13:40:36.034740 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:36.034748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:36.034907 (+ 159us) server_negotiation.cc:407] Connection header received
0926 13:40:36.034989 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:36.034992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:36.035045 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:36.035146 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:36.035761 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:36.036491 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:36.037377 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:36.037639 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:36.037777 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:36.041231 (+ 3454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:56153: BlockingRecv error: recv got EOF from 127.1.49.2:56153 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 13:40:36 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:36 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:36.258571 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:36.251963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:49169)
0926 13:40:36.252279 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:36.252293 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:36.252439 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:36.252823 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:36.252826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:36.253036 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:36.253219 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:36.253226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:36.254059 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:36.254062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:36.254844 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:36.254855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:36.255456 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:36.255460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:36.255533 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:36.256166 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:36.256179 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:36.258420 (+ 2241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":120,"threads_started":1}
W20260926 13:40:36.258793 10385 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:36.252009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:49169 (local address 127.1.49.62:33833)
0926 13:40:36.252170 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:36.252174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:36.252383 (+ 209us) server_negotiation.cc:407] Connection header received
0926 13:40:36.252601 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:36.252605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:36.252662 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:36.252764 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:36.253346 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:36.253933 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:36.254982 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:36.255208 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:36.255393 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:36.258638 (+ 3245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:49169: BlockingRecv error: recv got EOF from 127.1.49.3:49169 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 13:40:36 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:36 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:36.846326 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:36.829407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:44535)
0926 13:40:36.829753 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:36.829769 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:36.829890 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:36.830175 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:36.830177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:36.830375 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:36.830473 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:36.830479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:36.831519 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:36.831522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:36.832470 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:36.832485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:36.833200 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:36.833203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:36.833328 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:36.843404 (+ 10076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:36.843435 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:36.846127 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
W20260926 13:40:36.846539 10389 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:36.829420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:44535 (local address 127.1.49.62:33833)
0926 13:40:36.829694 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:36.829698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:36.829884 (+ 186us) server_negotiation.cc:407] Connection header received
0926 13:40:36.829970 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:36.829972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:36.830031 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:36.830124 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:36.830615 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:36.831313 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:36.832626 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:36.832939 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:36.833061 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:36.846336 (+ 13275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:44535: BlockingRecv error: recv got EOF from 127.1.49.1:44535 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
Sep 26 13:40:37 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:37 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:37.049448 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:37.042291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:38861)
0926 13:40:37.042932 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:37.042952 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:37.043078 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:37.043464 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:37.043468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:37.043690 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:37.043859 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:37.043866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:37.044790 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:37.044793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:37.045429 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:37.045439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:37.045989 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:37.045993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:37.046065 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:37.046740 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:37.046781 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:37.049304 (+ 2523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":89,"threads_started":1}
W20260926 13:40:37.049629 10389 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:37.042413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:38861 (local address 127.1.49.62:33833)
0926 13:40:37.042613 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:37.042617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:37.043040 (+ 423us) server_negotiation.cc:407] Connection header received
0926 13:40:37.043201 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:37.043204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:37.043254 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:37.043439 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:37.044016 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:37.044607 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:37.045571 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:37.045829 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:37.045925 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:37.049494 (+ 3569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:38861: BlockingRecv error: recv got EOF from 127.1.49.2:38861 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 13:40:37 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:37 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:37.265779 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:37.259401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:47943)
0926 13:40:37.259718 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:37.259733 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:37.259891 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:37.260190 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:37.260192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:37.260431 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:37.260552 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:37.260558 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:37.261382 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:37.261385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:37.261996 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:37.262005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:37.262488 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:37.262491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:37.262562 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:37.263152 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:37.263165 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:37.265597 (+ 2432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
W20260926 13:40:37.266000 10389 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:37.259570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:47943 (local address 127.1.49.62:33833)
0926 13:40:37.259772 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:37.259776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:37.259795 (+ 19us) server_negotiation.cc:407] Connection header received
0926 13:40:37.259999 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:37.260002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:37.260057 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:37.260159 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:37.260705 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:37.261271 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:37.262112 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:37.262363 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:37.262448 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:37.265807 (+ 3359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:47943: BlockingRecv error: recv got EOF from 127.1.49.3:47943 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 26 13:40:37 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:37 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:37.854575 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:37.847506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:37945)
0926 13:40:37.847910 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:37.847932 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:37.848070 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:37.848438 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:37.848441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:37.848706 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:37.848846 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:37.848854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:37.849986 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:37.849990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:37.850730 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:37.850741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:37.851259 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:37.851262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:37.851399 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:37.852024 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:37.852036 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:37.854415 (+ 2379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":96,"threads_started":1}
W20260926 13:40:37.854810 10393 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:37.847589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:37945 (local address 127.1.49.62:33833)
0926 13:40:37.847901 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:37.847908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:37.848048 (+ 140us) server_negotiation.cc:407] Connection header received
0926 13:40:37.848189 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:37.848193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:37.848261 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:37.848385 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:37.848990 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:37.849852 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:37.850886 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:37.851143 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:37.851240 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:37.854645 (+ 3405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:37945: BlockingRecv error: recv got EOF from 127.1.49.1:37945 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":98,"threads_started":1}
Sep 26 13:40:38 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:38 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:38.057052 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:38.050538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:54429)
0926 13:40:38.050821 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:38.050839 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:38.050945 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:38.051194 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:38.051196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:38.051449 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:38.051560 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:38.051566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:38.052654 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:38.052657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:38.053543 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:38.053555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:38.054118 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:38.054123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:38.054252 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:38.054889 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:38.054902 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:38.056926 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":123,"threads_started":1}
W20260926 13:40:38.057274 10393 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:38.050765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:54429 (local address 127.1.49.62:33833)
0926 13:40:38.050914 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:38.050917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:38.050936 (+ 19us) server_negotiation.cc:407] Connection header received
0926 13:40:38.051021 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:38.051024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:38.051066 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:38.051146 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:38.051699 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:38.052511 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:38.053698 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:38.053937 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:38.054045 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:38.057134 (+ 3089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:54429: BlockingRecv error: recv got EOF from 127.1.49.2:54429 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 13:40:38 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:38 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:38.274456 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:38.266975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:58581)
0926 13:40:38.267270 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:38.267289 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:38.267419 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:38.267848 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:38.267851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:38.268208 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:38.268354 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:38.268364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:38.269408 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:38.269411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:38.270302 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:38.270312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:38.270848 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:38.270851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:38.270917 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:38.271608 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:38.271623 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:38.274325 (+ 2702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":114,"threads_started":1}
W20260926 13:40:38.274742 10393 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:38.267076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:58581 (local address 127.1.49.62:33833)
0926 13:40:38.267309 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:38.267313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:38.267359 (+ 46us) server_negotiation.cc:407] Connection header received
0926 13:40:38.267593 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:38.267596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:38.267653 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:38.267746 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:38.268530 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:38.269257 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:38.270443 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:38.270711 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:38.270807 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:38.274603 (+ 3796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:58581: BlockingRecv error: recv got EOF from 127.1.49.3:58581 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 13:40:38 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:38 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:38.861739 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:38.855511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:35957)
0926 13:40:38.855944 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:38.855960 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:38.856075 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:38.856496 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:38.856499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:38.856684 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:38.856796 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:38.856802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:38.857700 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:38.857703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:38.858347 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:38.858364 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:38.858904 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:38.858907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:38.858976 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:38.859605 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:38.859618 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:38.861613 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":96,"threads_started":1}
W20260926 13:40:38.862052 10397 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:38.855701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:35957 (local address 127.1.49.62:33833)
0926 13:40:38.855983 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:38.855986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:38.856021 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:40:38.856285 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:38.856288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:38.856337 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:38.856424 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:38.856942 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:38.857568 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:38.858529 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:38.858766 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:38.858861 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:38.861884 (+ 3023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:35957: BlockingRecv error: recv got EOF from 127.1.49.1:35957 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":61,"threads_started":1}
Sep 26 13:40:39 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:39 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:39.065117 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:39.058019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:32931)
0926 13:40:39.058569 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:39.058591 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:39.058746 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:39.059045 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:39.059047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:39.059256 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:39.059415 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:39.059422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:39.060285 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:39.060290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:39.060882 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:39.060891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:39.061387 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:39.061390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:39.061481 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:39.062465 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:39.062486 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:39.064930 (+ 2444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":179,"threads_started":1}
W20260926 13:40:39.065196 10397 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:39.058289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:32931 (local address 127.1.49.62:33833)
0926 13:40:39.058527 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:39.058532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:39.058680 (+ 148us) server_negotiation.cc:407] Connection header received
0926 13:40:39.058864 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:39.058868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:39.058923 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:39.059018 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:39.059555 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:39.060169 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:39.061014 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:39.061257 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:39.061379 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:39.065082 (+ 3703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:32931: BlockingRecv error: recv got EOF from 127.1.49.2:32931 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Sep 26 13:40:39 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:39 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:39.281503 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:39.275249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:49793)
0926 13:40:39.275616 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:39.275636 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:39.275829 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:39.276136 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:39.276140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:39.276367 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:39.276464 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:39.276470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:39.277417 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:39.277424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:39.278055 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:39.278064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:39.278587 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:39.278591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:39.278692 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:39.279289 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:39.279301 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:39.281323 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":125,"threads_started":1}
W20260926 13:40:39.281598 10397 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:39.275436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:49793 (local address 127.1.49.62:33833)
0926 13:40:39.275608 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:39.275613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:39.275731 (+ 118us) server_negotiation.cc:407] Connection header received
0926 13:40:39.275948 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:39.275951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:39.276013 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:39.276127 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:39.276600 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:39.277287 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:39.278195 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:39.278449 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:39.278586 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:39.281476 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:49793: BlockingRecv error: recv got EOF from 127.1.49.3:49793 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 13:40:39 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:39 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:39.869127 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:39.862527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:44207)
0926 13:40:39.862873 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:39.862888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:39.862983 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:39.863288 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:39.863291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:39.863477 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:39.863573 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:39.863579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:39.864516 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:39.864519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:39.865114 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:39.865124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:39.865633 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:39.865636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:39.865709 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:39.866365 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:39.866378 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:39.868897 (+ 2519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":128,"threads_started":1}
W20260926 13:40:39.869200 10401 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:39.862648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:44207 (local address 127.1.49.62:33833)
0926 13:40:39.862894 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:39.862897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:39.862935 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:40:39.863067 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:39.863070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:39.863117 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:39.863204 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:39.863720 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:39.864389 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:39.865234 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:39.865495 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:39.865595 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:39.869058 (+ 3463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:44207: BlockingRecv error: recv got EOF from 127.1.49.1:44207 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
Sep 26 13:40:40 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:40 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:40.072669 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:40.066336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:43875)
0926 13:40:40.066679 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:40.066693 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:40.066811 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:40.067187 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:40.067193 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:40.067392 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:40.067537 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:40.067545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:40.068690 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:40.068707 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:40.069457 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:40.069469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:40.069955 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:40.069957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:40.070024 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:40.070606 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:40.070618 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:40.072542 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":102,"threads_started":1}
W20260926 13:40:40.073005 10401 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:40.066422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:43875 (local address 127.1.49.62:33833)
0926 13:40:40.066610 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:40.066615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:40.066785 (+ 170us) server_negotiation.cc:407] Connection header received
0926 13:40:40.066935 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:40.066938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:40.067005 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:40.067088 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:40.067715 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:40.068551 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:40.069579 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:40.069824 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:40.069915 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:40.072762 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:43875: BlockingRecv error: recv got EOF from 127.1.49.2:43875 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 13:40:40 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:40 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:40.289666 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:40.282626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:52027)
0926 13:40:40.282998 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:40.283014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:40.283141 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:40.283676 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:40.283679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:40.283917 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:40.284028 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:40.284035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:40.285055 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:40.285058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:40.285694 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:40.285704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:40.286293 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:40.286297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:40.286428 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:40.287132 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:40.287155 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:40.289539 (+ 2384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":133,"threads_started":1}
W20260926 13:40:40.289918 10401 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:40.282782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:52027 (local address 127.1.49.62:33833)
0926 13:40:40.282923 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:40.282928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:40.283097 (+ 169us) server_negotiation.cc:407] Connection header received
0926 13:40:40.283284 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:40.283288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:40.283347 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:40.283460 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:40.284267 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:40.284903 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:40.285844 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:40.286155 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:40.286271 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:40.289684 (+ 3413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:52027: BlockingRecv error: recv got EOF from 127.1.49.3:52027 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 13:40:40 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:40 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:40.877274 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:40.870213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:47873)
0926 13:40:40.870546 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:40.870560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:40.870658 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:40.870999 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:40.871001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:40.871203 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:40.871308 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:40.871314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:40.872460 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:40.872464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:40.873329 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:40.873344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:40.873896 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:40.873899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:40.873968 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:40.874665 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:40.874679 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:40.877045 (+ 2366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":129,"threads_started":1}
W20260926 13:40:40.877358 10405 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:40.870415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:47873 (local address 127.1.49.62:33833)
0926 13:40:40.870720 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:40.870724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:40.870741 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:40:40.870790 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:40.870792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:40.870838 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:40.870951 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:40.871474 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:40.872296 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:40.873499 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:40.873761 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:40.873861 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:40.877242 (+ 3381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:47873: BlockingRecv error: recv got EOF from 127.1.49.1:47873 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":129,"threads_started":1}
Sep 26 13:40:41 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:41 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:41.080476 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:41.073518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:35045)
0926 13:40:41.073920 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:41.073937 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:41.074117 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:41.074586 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:41.074589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:41.074830 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:41.074932 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:41.074939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:41.076030 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:41.076037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:41.076697 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:41.076732 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:41.077270 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:41.077275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:41.077374 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:41.078034 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:41.078048 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:41.080341 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":115,"threads_started":1}
W20260926 13:40:41.080792 10405 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:41.073680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:35045 (local address 127.1.49.62:33833)
0926 13:40:41.073892 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:41.073897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:41.074031 (+ 134us) server_negotiation.cc:407] Connection header received
0926 13:40:41.074373 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:41.074380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:41.074449 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:41.074558 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:41.075067 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:41.075907 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:41.076870 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:41.077130 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:41.077264 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:41.080524 (+ 3260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:35045: BlockingRecv error: recv got EOF from 127.1.49.2:35045 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 26 13:40:41 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:41 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:40:41.299862 10405 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:41.290829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:55183 (local address 127.1.49.62:33833)
0926 13:40:41.290988 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:41.290992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:41.291355 (+ 363us) server_negotiation.cc:407] Connection header received
0926 13:40:41.291526 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:41.291529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:41.291589 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:41.291686 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:41.292187 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:41.292961 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:41.295617 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:41.296111 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:41.296254 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:41.299683 (+ 3429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:55183: BlockingRecv error: recv got EOF from 127.1.49.3:55183 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 13:40:41.300189 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:41.290655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:55183)
0926 13:40:41.291236 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:41.291252 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:41.291408 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:41.291712 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:41.291715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:41.291959 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:41.292051 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:41.292057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:41.293120 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:41.293126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:41.294143 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:41.294154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:41.296435 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:41.296441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:41.296570 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:41.297242 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:41.297263 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:41.300057 (+ 2794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":233,"threads_started":1}
Sep 26 13:40:41 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:41 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:41.890923 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:41.878252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:38187)
0926 13:40:41.878555 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:41.878570 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:41.878671 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:41.878945 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:41.878947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:41.879138 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:41.879236 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:41.879241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:41.880368 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:41.880371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:41.881068 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:41.881080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:41.881633 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:41.881637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:41.881705 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:41.888487 (+ 6782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:41.888502 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:41.890743 (+ 2241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":121,"threads_started":1}
W20260926 13:40:41.891036 10409 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:41.878340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:38187 (local address 127.1.49.62:33833)
0926 13:40:41.878646 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:41.878650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:41.878667 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:40:41.878761 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:41.878763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:41.878810 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:41.878904 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:41.879524 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:41.880196 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:41.881252 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:41.881500 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:41.881604 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:41.890895 (+ 9291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:38187: BlockingRecv error: recv got EOF from 127.1.49.1:38187 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":133,"threads_started":1}
Sep 26 13:40:42 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:42 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:42.088269 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:42.081459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:45107)
0926 13:40:42.081767 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:42.081783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:42.081897 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:42.082191 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:42.082196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:42.082466 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:42.082581 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:42.082588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:42.083416 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:42.083419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:42.084154 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:42.084165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:42.084788 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:42.084792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:42.084878 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:42.085561 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:42.085574 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:42.088053 (+ 2479us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":106,"threads_started":1}
W20260926 13:40:42.088443 10409 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:42.081549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:45107 (local address 127.1.49.62:33833)
0926 13:40:42.081806 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:42.081810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:42.081844 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:40:42.081989 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:42.081992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:42.082036 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:42.082117 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:42.082708 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:42.083272 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:42.084310 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:42.084595 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:42.084699 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:42.088237 (+ 3538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:45107: BlockingRecv error: recv got EOF from 127.1.49.2:45107 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 26 13:40:42 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:42 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:42.313433 10411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:42.302611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:59555)
0926 13:40:42.302901 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:42.302917 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:42.303046 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:42.303411 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:42.303414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:42.303623 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:42.303745 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:42.303751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:42.304563 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:42.304568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:42.305141 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:42.305149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:42.305640 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:42.305644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:42.305737 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:42.310794 (+ 5057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:42.310810 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:42.313215 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
W20260926 13:40:42.313513 10409 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:42.302684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:59555 (local address 127.1.49.62:33833)
0926 13:40:42.302819 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:42.302824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:42.303003 (+ 179us) server_negotiation.cc:407] Connection header received
0926 13:40:42.303240 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:42.303243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:42.303292 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:42.303388 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:42.303907 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:42.304447 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:42.305260 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:42.305521 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:42.305635 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:42.313386 (+ 7751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:59555: BlockingRecv error: recv got EOF from 127.1.49.3:59555 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 13:40:42 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:42 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:42.897850 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:42.891943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:39949)
0926 13:40:42.892328 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:42.892344 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:42.892446 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:42.892792 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:42.892795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:42.892992 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:42.893164 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:42.893171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:42.893966 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:42.893969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:42.894549 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:42.894557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:42.895044 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:42.895046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:42.895112 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:42.895771 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:42.895784 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:42.897707 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":169,"threads_started":1}
W20260926 13:40:42.898100 10413 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:42.891999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:39949 (local address 127.1.49.62:33833)
0926 13:40:42.892335 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:42.892340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:42.892404 (+ 64us) server_negotiation.cc:407] Connection header received
0926 13:40:42.892573 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:42.892576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:42.892624 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:42.892706 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:42.893292 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:42.893844 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:42.894682 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:42.894927 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:42.895027 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:42.897954 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:39949: BlockingRecv error: recv got EOF from 127.1.49.1:39949 (error 108)
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":124,"threads_started":1}
Sep 26 13:40:43 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:43 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:43.096365 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:43.089500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:47037)
0926 13:40:43.089824 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:43.089839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:43.089953 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:43.090230 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:43.090233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:43.090437 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:43.090535 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:43.090541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:43.091486 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:43.091489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:43.092191 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:43.092201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:43.092820 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:43.092824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:43.092903 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:43.093670 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:43.093686 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:43.096174 (+ 2488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":73,"threads_started":1}
W20260926 13:40:43.096424 10413 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:43.089693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:47037 (local address 127.1.49.62:33833)
0926 13:40:43.089831 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:43.089835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:43.089904 (+ 69us) server_negotiation.cc:407] Connection header received
0926 13:40:43.090033 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:43.090035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:43.090076 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:43.090158 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:43.090669 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:43.091209 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:43.092379 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:43.092653 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:43.092760 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:43.096315 (+ 3555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:47037: BlockingRecv error: recv got EOF from 127.1.49.2:47037 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 13:40:43 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:43 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:43.321031 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:43.314501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:44121)
0926 13:40:43.314797 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:43.314818 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:43.314933 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:43.315210 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:43.315212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:43.315429 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:43.315529 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:43.315535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:43.316536 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:43.316538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:43.317192 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:43.317203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:43.317859 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:43.317863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:43.317949 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:43.318630 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:43.318643 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:43.320901 (+ 2258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":127,"threads_started":1}
W20260926 13:40:43.321164 10413 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:43.314709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:44121 (local address 127.1.49.62:33833)
0926 13:40:43.314865 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:43.314869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:43.314885 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:40:43.315030 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:43.315033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:43.315082 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:43.315169 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:43.315677 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:43.316390 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:43.317363 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:43.317685 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:43.317806 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:43.321060 (+ 3254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:44121: BlockingRecv error: recv got EOF from 127.1.49.3:44121 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 13:40:43 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:43 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:43.904731 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:43.898832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:36367)
0926 13:40:43.899133 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:43.899147 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:43.899243 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:43.899513 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:43.899515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:43.899694 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:43.899903 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:43.899911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:43.900687 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:43.900689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:43.901269 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:43.901277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:43.901699 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:43.901724 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:43.901789 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:43.902311 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:43.902322 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:43.904561 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":121,"threads_started":1}
W20260926 13:40:43.904927 10417 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:43.898972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:36367 (local address 127.1.49.62:33833)
0926 13:40:43.899211 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:43.899215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:43.899229 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:40:43.899337 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:43.899339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:43.899386 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:43.899476 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:43.900025 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:43.900585 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:43.901375 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:43.901597 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:43.901678 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:43.904790 (+ 3112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:36367: BlockingRecv error: recv got EOF from 127.1.49.1:36367 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
Sep 26 13:40:44 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:44 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:44.104107 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:44.097392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:54277)
0926 13:40:44.097865 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:44.097882 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:44.098044 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:44.098377 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:44.098380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:44.098589 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:44.098711 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:44.098717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:44.099596 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:44.099600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:44.100616 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:44.100629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:44.101254 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:44.101257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:44.101324 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:44.101876 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:44.101887 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:44.103955 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":112,"threads_started":1}
W20260926 13:40:44.104346 10417 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:44.097555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:54277 (local address 127.1.49.62:33833)
0926 13:40:44.097868 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:44.097875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:44.097977 (+ 102us) server_negotiation.cc:407] Connection header received
0926 13:40:44.098151 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:44.098155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:44.098212 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:44.098307 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:44.098868 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:44.099465 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:44.100837 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:44.101121 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:44.101211 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:44.104222 (+ 3011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:54277: BlockingRecv error: recv got EOF from 127.1.49.2:54277 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
Sep 26 13:40:44 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:44 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:44.328465 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:44.322247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:53869)
0926 13:40:44.322536 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:44.322549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:44.322653 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:44.322910 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:44.322912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:44.323124 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:44.323235 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:44.323242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:44.324236 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:44.324239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:44.324825 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:44.324834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:44.325366 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:44.325369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:44.325441 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:44.325981 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:44.325993 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:44.328318 (+ 2325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":108,"threads_started":1}
W20260926 13:40:44.328645 10417 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:44.322357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:53869 (local address 127.1.49.62:33833)
0926 13:40:44.322517 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:44.322521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:44.322641 (+ 120us) server_negotiation.cc:407] Connection header received
0926 13:40:44.322745 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:44.322747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:44.322795 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:44.322870 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:44.323404 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:44.324061 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:44.324955 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:44.325176 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:44.325323 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:44.328493 (+ 3170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:53869: BlockingRecv error: recv got EOF from 127.1.49.3:53869 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 13:40:44 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:44 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:44.912461 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:44.905804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:37733)
0926 13:40:44.906069 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:44.906092 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:44.906249 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:44.906583 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:44.906589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:44.906896 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:44.907006 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:44.907013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:44.908041 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:44.908045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:44.909034 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:44.909045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:44.909527 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:44.909531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:44.909622 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:44.910151 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:44.910164 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:44.912302 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":95,"threads_started":1}
W20260926 13:40:44.912647 10421 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:44.905886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:37733 (local address 127.1.49.62:33833)
0926 13:40:44.906208 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:44.906212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:44.906229 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:40:44.906372 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:44.906375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:44.906431 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:44.906526 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:44.907218 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:44.907926 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:44.909159 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:44.909404 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:44.909521 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:44.912502 (+ 2981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:37733: BlockingRecv error: recv got EOF from 127.1.49.1:37733 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":107,"threads_started":1}
Sep 26 13:40:45 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:45 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:45.111676 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:45.105089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:37027)
0926 13:40:45.105393 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:45.105458 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:45.105567 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:45.105905 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:45.105908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:45.106188 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:45.106363 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:45.106369 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:45.107286 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:45.107289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:45.108220 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:45.108230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:45.108777 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:45.108803 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:45.108966 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:45.109657 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:45.109670 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:45.111545 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":117,"threads_started":1}
W20260926 13:40:45.112032 10421 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:45.105298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:37027 (local address 127.1.49.62:33833)
0926 13:40:45.105518 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:45.105525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:45.105544 (+ 19us) server_negotiation.cc:407] Connection header received
0926 13:40:45.105681 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:45.105684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:45.105746 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:45.105867 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:45.106454 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:45.107029 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:45.108321 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:45.108607 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:45.108707 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:45.111885 (+ 3178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:37027: BlockingRecv error: recv got EOF from 127.1.49.2:37027 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 13:40:45 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:45 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:45.336077 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:45.329522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:43511)
0926 13:40:45.329939 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:45.329977 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:45.330097 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:45.330382 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:45.330385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:45.330569 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:45.330677 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:45.330683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:45.331686 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:45.331689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:45.332338 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:45.332352 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:45.332821 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:45.332824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:45.332890 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:45.333475 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:45.333489 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:45.335925 (+ 2436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":110,"threads_started":1}
W20260926 13:40:45.336192 10421 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:45.329590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:43511 (local address 127.1.49.62:33833)
0926 13:40:45.329758 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:45.329763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:45.330111 (+ 348us) server_negotiation.cc:407] Connection header received
0926 13:40:45.330183 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:45.330185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:45.330241 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:45.330318 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:45.330807 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:45.331467 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:45.332444 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:45.332705 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:45.332800 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:45.336075 (+ 3275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:43511: BlockingRecv error: recv got EOF from 127.1.49.3:43511 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 13:40:45 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:45 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:45.919230 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:45.913332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:57101)
0926 13:40:45.913658 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:45.913676 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:45.913793 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:45.914073 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:45.914076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:45.914266 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:45.914409 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:45.914415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:45.915341 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:45.915344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:45.915960 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:45.915971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:45.916481 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:45.916484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:45.916550 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:45.917125 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:45.917137 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:45.919106 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":110,"threads_started":1}
W20260926 13:40:45.919451 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:45.913443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:57101 (local address 127.1.49.62:33833)
0926 13:40:45.913714 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:45.913721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:45.913740 (+ 19us) server_negotiation.cc:407] Connection header received
0926 13:40:45.913903 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:45.913905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:45.913952 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:45.914040 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:45.914552 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:45.915164 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:45.916094 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:45.916362 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:45.916462 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:45.919298 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:57101: BlockingRecv error: recv got EOF from 127.1.49.1:57101 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
Sep 26 13:40:46 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:46 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:46.118825 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:46.112603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:36297)
0926 13:40:46.112961 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:46.112979 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:46.113150 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:46.113450 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:46.113453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:46.113702 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:46.113802 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:46.113809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:46.114619 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:46.114627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:46.115211 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:46.115220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:46.115783 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:46.115787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:46.115918 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:46.116569 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:46.116582 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:46.118625 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":109,"threads_started":1}
W20260926 13:40:46.118935 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:46.112770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:36297 (local address 127.1.49.62:33833)
0926 13:40:46.112954 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:46.112960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:46.113150 (+ 190us) server_negotiation.cc:407] Connection header received
0926 13:40:46.113260 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:46.113265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:46.113325 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:46.113424 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:46.113933 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:46.114494 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:46.115347 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:46.115644 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:46.115778 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:46.118799 (+ 3021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:36297: BlockingRecv error: recv got EOF from 127.1.49.2:36297 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 13:40:46 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:46 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:46.344265 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:46.337137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:43169)
0926 13:40:46.337471 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:46.337486 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:46.337602 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:46.338014 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:46.338017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:46.338286 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:46.338419 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:46.338426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:46.339342 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:46.339345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:46.340009 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:46.340019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:46.340637 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:46.340640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:46.340726 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:46.341483 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:46.341504 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:46.344125 (+ 2621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":140,"threads_started":1}
W20260926 13:40:46.344626 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:46.337355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:43169 (local address 127.1.49.62:33833)
0926 13:40:46.337507 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:46.337511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:46.337546 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:40:46.337767 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:46.337771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:46.337827 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:46.337918 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:46.338585 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:46.339196 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:46.340215 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:46.340483 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:46.340588 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:46.344283 (+ 3695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:43169: BlockingRecv error: recv got EOF from 127.1.49.3:43169 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 13:40:46 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:46 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:46.931183 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:46.920181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:40215)
0926 13:40:46.920434 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:46.920448 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:46.920569 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:46.920870 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:46.920873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:46.921127 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:46.921274 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:46.921285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:46.922177 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:46.922183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:46.923086 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:46.923098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:46.923851 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:46.923855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:46.923939 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:46.928676 (+ 4737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:46.928700 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:46.930992 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":87,"threads_started":1}
W20260926 13:40:46.931432 10429 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:46.920227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:40215 (local address 127.1.49.62:33833)
0926 13:40:46.920513 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:46.920517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:46.920533 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:40:46.920659 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:46.920662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:46.920732 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:46.920832 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:46.921399 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:46.922004 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:46.923247 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:46.923586 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:46.923681 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:46.931285 (+ 7604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:40215: BlockingRecv error: recv got EOF from 127.1.49.1:40215 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":78,"threads_started":1}
Sep 26 13:40:47 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:47 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:47.126017 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:47.119868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:55549)
0926 13:40:47.120167 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:47.120187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:47.120334 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:47.120633 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:47.120636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:47.120877 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:47.120975 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:47.120981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:47.121844 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:47.121848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:47.122416 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:47.122424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:47.122901 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:47.122904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:47.122996 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:47.123591 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:47.123603 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:47.125861 (+ 2258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":85,"threads_started":1}
W20260926 13:40:47.126181 10429 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:47.119936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:55549 (local address 127.1.49.62:33833)
0926 13:40:47.120130 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:47.120135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:47.120269 (+ 134us) server_negotiation.cc:407] Connection header received
0926 13:40:47.120447 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:47.120451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:47.120509 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:47.120597 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:47.121100 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:47.121717 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:47.122539 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:47.122776 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:47.122892 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:47.126047 (+ 3155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:55549: BlockingRecv error: recv got EOF from 127.1.49.2:55549 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 26 13:40:47 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:47 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.3@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:47.351653 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:47.345178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.3:44493)
0926 13:40:47.345477 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:47.345492 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:47.345602 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:47.345884 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:47.345886 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:47.346083 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:47.346248 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:47.346254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:47.347205 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:47.347208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:47.347964 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:47.347976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:47.348538 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:47.348540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:47.348601 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:47.349160 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:47.349172 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:47.351450 (+ 2278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":103,"threads_started":1}
W20260926 13:40:47.351788 10429 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:47.345296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:44493 (local address 127.1.49.62:33833)
0926 13:40:47.345462 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:47.345466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:47.345603 (+ 137us) server_negotiation.cc:407] Connection header received
0926 13:40:47.345686 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:47.345688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:47.345751 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:47.345828 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:47.346421 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:47.347056 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:47.348152 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:47.348426 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:47.348520 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:47.351657 (+ 3137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.3:44493: BlockingRecv error: recv got EOF from 127.1.49.3:44493 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 13:40:47 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:47 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.1@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:40:47.939046 10433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:47.932645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:48559 (local address 127.1.49.62:33833)
0926 13:40:47.932955 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:47.932961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:47.932976 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:40:47.933111 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:47.933114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:47.933171 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:47.933268 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:47.933837 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:47.934486 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:47.935567 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:47.935862 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:47.935958 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:47.938930 (+ 2972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.1:48559: BlockingRecv error: recv got EOF from 127.1.49.1:48559 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260926 13:40:47.938855 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:47.932542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.1:48559)
0926 13:40:47.932869 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:47.932887 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:47.933004 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:47.933327 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:47.933333 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:47.933567 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:47.933666 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:47.933672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:47.934625 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:47.934628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:47.935248 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:47.935259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:47.936002 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:47.936005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:47.936082 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:47.936638 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:47.936679 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:47.938729 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":107,"threads_started":1}
Sep 26 13:40:48 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:40:48 dist-test-slave-1wns krb5kdc[9375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.49.2@KRBTEST.COM for oryx/127.1.49.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:40:48.134582 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:48.127424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33833 (local address 127.1.49.2:59475)
0926 13:40:48.127746 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:48.127761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:48.127899 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:48.128185 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:48.128188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:48.128385 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:48.128514 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:48.128520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:48.129409 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:48.129412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:48.130096 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:48.130106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:48.131001 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:48.131005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:48.131116 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:48.132065 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:48.132083 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:48.134436 (+ 2353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.49.62:33833: Server oryx/127.1.49.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":117,"threads_started":1}
W20260926 13:40:48.134867 10433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:48.127546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:59475 (local address 127.1.49.62:33833)
0926 13:40:48.127733 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:48.127736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:48.127839 (+ 103us) server_negotiation.cc:407] Connection header received
0926 13:40:48.127989 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:48.127992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:48.128041 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:48.128126 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:48.128645 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:48.129202 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:48.130317 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:48.130821 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:48.130952 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:48.134731 (+ 3779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.49.2:59475: BlockingRecv error: recv got EOF from 127.1.49.2:59475 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 13:40:48.217310 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 9958
I20260926 13:40:48.223392 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 10110
I20260926 13:40:48.229055 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 10243
I20260926 13:40:48.234771 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 9888
2026-09-26T13:40:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18276 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T13:40:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:48Z Disabled control of system clock
I20260926 13:40:48.268024 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:44611
--webserver_interface=127.1.49.62
--webserver_port=0
--builtin_ntp_servers=127.1.49.20:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:44611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 13:40:48.399081 10438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:48.399483 10438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:48.399526 10438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:48.404102 10438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:48.404265 10438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:48.404297 10438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:48.404331 10438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:48.404348 10438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:48.409725 10438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:44611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:44611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:48.410948 10438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:48.412505 10438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:48.422703 10443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:48.422825 10444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:48.422688 10446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:48.423007 10438 server_base.cc:1034] running on GCE node
I20260926 13:40:48.423734 10438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:48.425135 10438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:48.426362 10438 hybrid_clock.cc:648] HybridClock initialized: now 1790430048426346 us; error 40 us; skew 500 ppm
I20260926 13:40:48.428866 10438 webserver.cc:460] Webserver started at http://127.1.49.62:45177/ using document root <none> and password file <none>
I20260926 13:40:48.429548 10438 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:48.429652 10438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:48.429932 10438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:48.432297 10438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "2f1371934614484ea61c84d941844001"
format_stamp: "Formatted at 2026-09-26 13:40:48 on dist-test-slave-1wns"
server_key: "6fba7669cd84e8450fb6cd4569a0aaf1"
server_key_iv: "e20c2abe19bb4f9d820282e58357246a"
server_key_version: "encryptionkey@0"
I20260926 13:40:48.432946 10438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "2f1371934614484ea61c84d941844001"
format_stamp: "Formatted at 2026-09-26 13:40:48 on dist-test-slave-1wns"
server_key: "6fba7669cd84e8450fb6cd4569a0aaf1"
server_key_iv: "e20c2abe19bb4f9d820282e58357246a"
server_key_version: "encryptionkey@0"
I20260926 13:40:48.437171 10438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:40:48.440102 10452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:48.441787 10438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:48.441979 10438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "2f1371934614484ea61c84d941844001"
format_stamp: "Formatted at 2026-09-26 13:40:48 on dist-test-slave-1wns"
server_key: "6fba7669cd84e8450fb6cd4569a0aaf1"
server_key_iv: "e20c2abe19bb4f9d820282e58357246a"
server_key_version: "encryptionkey@0"
I20260926 13:40:48.442138 10438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:48.454162 10438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:48.455061 10438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:48.455305 10438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:48.463274 10438 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:44611
I20260926 13:40:48.463330 10504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:44611 every 8 connection(s)
I20260926 13:40:48.464515 10438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:48.465592 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 10438
I20260926 13:40:48.465770 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:48.466081 1220 external_mini_cluster.cc:1428] Setting key 45905c43e7aec26f259ce76f438a80db
I20260926 13:40:48.468950 10505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:48.476209 10505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001: Bootstrap starting.
I20260926 13:40:48.477010 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:48.467747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.1.49.62:44611)
0926 13:40:48.468750 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:48.468769 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:48.468812 (+ 43us) server_negotiation.cc:407] Connection header received
0926 13:40:48.469591 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:48.469612 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:48.470170 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:48.470644 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:48.471150 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:48.472274 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:48.473247 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:48.473775 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:48.473985 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:48.474976 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:48.474981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:48.474996 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:48.475456 (+ 460us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:48.475522 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:48.475544 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:48.475664 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:48.475792 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 13:40:48.476142 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":158,"threads_started":1}
I20260926 13:40:48.479074 10505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:48.480162 10505 log.cc:824] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:48.482563 10505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001: No bootstrap required, opened a new log
I20260926 13:40:48.486959 10505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1371934614484ea61c84d941844001" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44611 } }
I20260926 13:40:48.487280 10505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:48.487322 10505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1371934614484ea61c84d941844001, State: Initialized, Role: FOLLOWER
I20260926 13:40:48.487754 10505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1371934614484ea61c84d941844001" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44611 } }
I20260926 13:40:48.487960 10505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:48.488014 10505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:48.488111 10505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:48.489217 10505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1371934614484ea61c84d941844001" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44611 } }
I20260926 13:40:48.489656 10505 leader_election.cc:302] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f1371934614484ea61c84d941844001; no voters:
I20260926 13:40:48.490146 10505 leader_election.cc:288] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:48.490433 10510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:48.490790 10510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [term 1 LEADER]: Becoming Leader. State: Replica: 2f1371934614484ea61c84d941844001, State: Running, Role: LEADER
I20260926 13:40:48.491137 10510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1371934614484ea61c84d941844001" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44611 } }
I20260926 13:40:48.491653 10505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:48.493110 10512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f1371934614484ea61c84d941844001. Latest consensus state: current_term: 1 leader_uuid: "2f1371934614484ea61c84d941844001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1371934614484ea61c84d941844001" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44611 } } }
I20260926 13:40:48.493110 10511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f1371934614484ea61c84d941844001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1371934614484ea61c84d941844001" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44611 } } }
I20260926 13:40:48.493297 10512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:48.493324 10511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:48.493732 10519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:48.497210 10519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:48.503756 10519 catalog_manager.cc:1347] Generated new cluster ID: a926b869e2d54b0c904807e442c7ffc4
I20260926 13:40:48.503943 10519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:48.524359 10519 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:48.525710 10519 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:48.531684 10519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f1371934614484ea61c84d941844001: Generated new TSK 0
I20260926 13:40:48.532662 10519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:40:48.537719 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:44611
--builtin_ntp_servers=127.1.49.20:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 13:40:48.661792 10529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:48.662124 10529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:48.662171 10529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:48.666601 10529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:48.666687 10529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:48.666747 10529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:40:48.671880 10529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:44611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:48.673098 10529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:48.674656 10529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:48.685786 10534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:48.685773 10537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:48.685774 10535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:48.685817 10529 server_base.cc:1034] running on GCE node
I20260926 13:40:48.686581 10529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:48.687480 10529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:48.688694 10529 hybrid_clock.cc:648] HybridClock initialized: now 1790430048688645 us; error 63 us; skew 500 ppm
I20260926 13:40:48.691102 10529 webserver.cc:460] Webserver started at http://127.1.49.1:41525/ using document root <none> and password file <none>
I20260926 13:40:48.691895 10529 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:48.691979 10529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:48.692292 10529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:48.695067 10529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "9fffb4fa268a4aa4aa46d5890a852438"
format_stamp: "Formatted at 2026-09-26 13:40:48 on dist-test-slave-1wns"
server_key: "16aa8e0ff60e651d270884e1d00f0faf"
server_key_iv: "e8682a45374da16ce1ea517f008d8b24"
server_key_version: "encryptionkey@0"
I20260926 13:40:48.695752 10529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "9fffb4fa268a4aa4aa46d5890a852438"
format_stamp: "Formatted at 2026-09-26 13:40:48 on dist-test-slave-1wns"
server_key: "16aa8e0ff60e651d270884e1d00f0faf"
server_key_iv: "e8682a45374da16ce1ea517f008d8b24"
server_key_version: "encryptionkey@0"
I20260926 13:40:48.700651 10529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 13:40:48.703959 10543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:48.705607 10529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:40:48.705767 10529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "9fffb4fa268a4aa4aa46d5890a852438"
format_stamp: "Formatted at 2026-09-26 13:40:48 on dist-test-slave-1wns"
server_key: "16aa8e0ff60e651d270884e1d00f0faf"
server_key_iv: "e8682a45374da16ce1ea517f008d8b24"
server_key_version: "encryptionkey@0"
I20260926 13:40:48.705909 10529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:48.718364 10529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:48.719249 10529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:48.719470 10529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:48.720191 10529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:48.721268 10529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:48.721351 10529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:48.721415 10529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:48.721467 10529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:48.731442 10529 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:42439
I20260926 13:40:48.731485 10656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:42439 every 8 connection(s)
I20260926 13:40:48.732504 10529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:40:48.735410 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 10529
I20260926 13:40:48.735569 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:40:48.735975 1220 external_mini_cluster.cc:1428] Setting key 3c80a425dc244f370d22aecbfa252585
I20260926 13:40:48.738344 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:44611
--builtin_ntp_servers=127.1.49.20:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 13:40:48.743587 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:48.734583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:50079 (local address 127.1.49.62:44611)
0926 13:40:48.734773 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:48.734778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:48.736121 (+ 1343us) server_negotiation.cc:407] Connection header received
0926 13:40:48.737980 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:48.737984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:48.738045 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:48.738128 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:48.739260 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:48.739942 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:48.741270 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:48.741525 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:48.741657 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:48.742992 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:48.742996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:48.743001 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:48.743024 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:48.743045 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:48.743053 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:48.743147 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:48.743334 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 13:40:48.743421 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 13:40:48.744650 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:48.735055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:44611 (local address 127.1.49.1:50079)
0926 13:40:48.735934 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:48.735995 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:48.737728 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:48.738337 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:48.738346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:48.738449 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:48.739094 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:48.739110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:48.740092 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:48.740102 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:48.741121 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:48.741132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:48.741721 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:48.741726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:48.741922 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:48.742679 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:40:48.742681 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:48.742727 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:40:48.742804 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:48.743155 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:48.743167 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:48.743175 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 13:40:48.743334 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 13:40:48.743690 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":155,"threads_started":1}
I20260926 13:40:48.746209 10657 heartbeater.cc:347] Connected to a master server at 127.1.49.62:44611
I20260926 13:40:48.746516 10657 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:48.747351 10657 heartbeater.cc:511] Master 127.1.49.62:44611 requested a full tablet report, sending...
I20260926 13:40:48.749533 10468 ts_manager.cc:194] Registered new tserver with Master: 9fffb4fa268a4aa4aa46d5890a852438 (127.1.49.1:42439)
I20260926 13:40:48.750885 10468 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.49.1:50079
W20260926 13:40:48.858079 10660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:48.858412 10660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:48.858453 10660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:48.862891 10660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:48.862984 10660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:48.863045 10660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:40:48.868402 10660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:44611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:48.869720 10660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:48.871294 10660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:48.881981 10665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:48.881996 10668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:48.882485 10666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:48.882931 10660 server_base.cc:1034] running on GCE node
I20260926 13:40:48.883417 10660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:48.884166 10660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:48.885526 10660 hybrid_clock.cc:648] HybridClock initialized: now 1790430048885500 us; error 49 us; skew 500 ppm
I20260926 13:40:48.888145 10660 webserver.cc:460] Webserver started at http://127.1.49.2:38913/ using document root <none> and password file <none>
I20260926 13:40:48.888818 10660 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:48.888922 10660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:48.889185 10660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:48.891674 10660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "aac87c5156c44281b5eb2f1746739acf"
format_stamp: "Formatted at 2026-09-26 13:40:48 on dist-test-slave-1wns"
server_key: "b04227edf1ce6ea47ed83ec78aecde6e"
server_key_iv: "b3a460659895956f13ee4512d9af37a0"
server_key_version: "encryptionkey@0"
I20260926 13:40:48.892423 10660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "aac87c5156c44281b5eb2f1746739acf"
format_stamp: "Formatted at 2026-09-26 13:40:48 on dist-test-slave-1wns"
server_key: "b04227edf1ce6ea47ed83ec78aecde6e"
server_key_iv: "b3a460659895956f13ee4512d9af37a0"
server_key_version: "encryptionkey@0"
I20260926 13:40:48.897104 10660 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:40:48.900492 10674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:48.901965 10660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:40:48.902143 10660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "aac87c5156c44281b5eb2f1746739acf"
format_stamp: "Formatted at 2026-09-26 13:40:48 on dist-test-slave-1wns"
server_key: "b04227edf1ce6ea47ed83ec78aecde6e"
server_key_iv: "b3a460659895956f13ee4512d9af37a0"
server_key_version: "encryptionkey@0"
I20260926 13:40:48.902281 10660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:48.920717 10660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:48.921528 10660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:48.921757 10660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:48.922396 10660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:48.923659 10660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:48.923735 10660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:48.923832 10660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:48.923873 10660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:48.933648 10660 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:42009
I20260926 13:40:48.933702 10787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:42009 every 8 connection(s)
I20260926 13:40:48.934639 10660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:40:48.936043 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 10660
I20260926 13:40:48.936168 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:40:48.936506 1220 external_mini_cluster.cc:1428] Setting key 9a680dc7dbe4448e54f214eda0c6f444
I20260926 13:40:48.938905 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:44611
--builtin_ntp_servers=127.1.49.20:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 13:40:48.946978 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:48.936926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:33753 (local address 127.1.49.62:44611)
0926 13:40:48.937077 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:48.937084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:48.938475 (+ 1391us) server_negotiation.cc:407] Connection header received
0926 13:40:48.940108 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:48.940115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:48.940187 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:48.940301 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:48.941734 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:48.942341 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:48.944408 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:48.944649 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:48.944786 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:48.946413 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:48.946416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:48.946418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:48.946434 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:48.946454 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:48.946463 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:48.946527 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:48.946791 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 13:40:48.946855 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:40:48.948535 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:48.937486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:44611 (local address 127.1.49.2:33753)
0926 13:40:48.938298 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:48.938361 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:48.939856 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:48.940564 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:48.940578 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:48.940714 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:48.941512 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:48.941538 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:48.942499 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:48.942506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:48.944242 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:48.944259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:48.944863 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:48.944870 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:48.945147 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:48.945991 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:40:48.945995 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:48.946094 (+ 99us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:40:48.946207 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:48.946559 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:48.946571 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:48.946582 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 13:40:48.946801 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 13:40:48.947194 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":153,"threads_started":1}
I20260926 13:40:48.950165 10788 heartbeater.cc:347] Connected to a master server at 127.1.49.62:44611
I20260926 13:40:48.950484 10788 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:48.951117 10788 heartbeater.cc:511] Master 127.1.49.62:44611 requested a full tablet report, sending...
I20260926 13:40:48.952476 10468 ts_manager.cc:194] Registered new tserver with Master: aac87c5156c44281b5eb2f1746739acf (127.1.49.2:42009)
I20260926 13:40:48.953140 10468 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.49.2:33753
W20260926 13:40:49.063838 10791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:49.064245 10791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:49.064337 10791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:49.069094 10791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:49.069207 10791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:49.069304 10791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:40:49.074589 10791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:44611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:49.075919 10791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:49.077521 10791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:49.088089 10796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:49.088107 10797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:49.088120 10799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:49.088662 10791 server_base.cc:1034] running on GCE node
I20260926 13:40:49.089099 10791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:49.089820 10791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:49.091020 10791 hybrid_clock.cc:648] HybridClock initialized: now 1790430049090970 us; error 65 us; skew 500 ppm
I20260926 13:40:49.093400 10791 webserver.cc:460] Webserver started at http://127.1.49.3:38061/ using document root <none> and password file <none>
I20260926 13:40:49.094099 10791 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:49.094218 10791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:49.094465 10791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:49.096740 10791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "e8744b5e18da4c9ea19e670f2ede6175"
format_stamp: "Formatted at 2026-09-26 13:40:49 on dist-test-slave-1wns"
server_key: "799da602a3476b80098094eed03866ad"
server_key_iv: "d57efe3f06cc1749d4e8940e79fca369"
server_key_version: "encryptionkey@0"
I20260926 13:40:49.097340 10791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "e8744b5e18da4c9ea19e670f2ede6175"
format_stamp: "Formatted at 2026-09-26 13:40:49 on dist-test-slave-1wns"
server_key: "799da602a3476b80098094eed03866ad"
server_key_iv: "d57efe3f06cc1749d4e8940e79fca369"
server_key_version: "encryptionkey@0"
I20260926 13:40:49.101864 10791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:40:49.105106 10805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:49.106873 10791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:40:49.107050 10791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "e8744b5e18da4c9ea19e670f2ede6175"
format_stamp: "Formatted at 2026-09-26 13:40:49 on dist-test-slave-1wns"
server_key: "799da602a3476b80098094eed03866ad"
server_key_iv: "d57efe3f06cc1749d4e8940e79fca369"
server_key_version: "encryptionkey@0"
I20260926 13:40:49.107230 10791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:49.121420 10791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:49.122184 10791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:49.122387 10791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:49.123138 10791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:49.124316 10791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:49.124400 10791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:49.124478 10791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:49.124533 10791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:49.134598 10791 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:43863
I20260926 13:40:49.134617 10918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:43863 every 8 connection(s)
I20260926 13:40:49.135682 10791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:40:49.136803 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 10791
I20260926 13:40:49.136922 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:40:49.137233 1220 external_mini_cluster.cc:1428] Setting key 53b78c28896d41aa23aabec4fa124c87
I20260926 13:40:49.146422 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:49.138030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:56631 (local address 127.1.49.62:44611)
0926 13:40:49.138182 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:49.138186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:49.139787 (+ 1601us) server_negotiation.cc:407] Connection header received
0926 13:40:49.141295 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:49.141299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:49.141358 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:49.141438 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:49.142437 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:49.142983 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:49.144299 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:49.144529 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:49.144645 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:49.145864 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:49.145867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:49.145869 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:49.145885 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:49.145905 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:49.145913 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:49.145997 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:49.146200 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 13:40:49.146269 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:40:49.147490 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:49.138709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:44611 (local address 127.1.49.3:56631)
0926 13:40:49.139621 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:49.139680 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:49.141046 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:49.141642 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:49.141650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:49.141748 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:49.142287 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:49.142301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:49.143127 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:49.143131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:49.144164 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:49.144175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:49.144711 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:49.144718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:49.144888 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:49.145557 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:40:49.145559 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:49.145603 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:40:49.145677 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:49.146017 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:49.146032 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:49.146037 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:40:49.146197 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 13:40:49.146530 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":133,"threads_started":1}
I20260926 13:40:49.148803 10919 heartbeater.cc:347] Connected to a master server at 127.1.49.62:44611
I20260926 13:40:49.149066 10919 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:49.149669 10919 heartbeater.cc:511] Master 127.1.49.62:44611 requested a full tablet report, sending...
I20260926 13:40:49.150880 10468 ts_manager.cc:194] Registered new tserver with Master: e8744b5e18da4c9ea19e670f2ede6175 (127.1.49.3:43863)
I20260926 13:40:49.151610 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:40:49.151633 10468 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.49.3:56631
W20260926 13:40:49.154829 10508 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:49.153832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54094 (local address 127.1.49.62:44611)
0926 13:40:49.153988 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:49.153993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:49.154149 (+ 156us) server_negotiation.cc:407] Connection header received
0926 13:40:49.154307 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:49.154310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:49.154361 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:49.154448 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:49.154694 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54094: BlockingRecv error: recv got EOF from 127.0.0.1:54094 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:40:49.155342 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 10529
I20260926 13:40:49.161259 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 10660
I20260926 13:40:49.167471 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 10791
I20260926 13:40:49.173202 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 10438
2026-09-26T13:40:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (937 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T13:40:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:49Z Disabled control of system clock
I20260926 13:40:49.205363 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:34997
--webserver_interface=127.1.49.62
--webserver_port=0
--builtin_ntp_servers=127.1.49.20:46195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:34997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 13:40:49.331231 10930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:49.331583 10930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:49.331624 10930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:49.336419 10930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:49.336534 10930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:49.336555 10930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:49.336576 10930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:49.336596 10930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:49.342031 10930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:46195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:34997
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:34997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:49.343223 10930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:49.344743 10930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:49.354532 10935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:49.354691 10930 server_base.cc:1034] running on GCE node
W20260926 13:40:49.354553 10936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:49.354514 10938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:49.355554 10930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:49.356875 10930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:49.358047 10930 hybrid_clock.cc:648] HybridClock initialized: now 1790430049358009 us; error 56 us; skew 500 ppm
I20260926 13:40:49.360354 10930 webserver.cc:460] Webserver started at http://127.1.49.62:44495/ using document root <none> and password file <none>
I20260926 13:40:49.361001 10930 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:49.361071 10930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:49.361304 10930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:49.363766 10930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "6abebc6c70774aca9ae022449650ca25"
format_stamp: "Formatted at 2026-09-26 13:40:49 on dist-test-slave-1wns"
server_key: "4e6a41a54960634ab205a899077d0fda"
server_key_iv: "2d2224bace52f5cc3b6ce3ee8e93ebc8"
server_key_version: "encryptionkey@0"
I20260926 13:40:49.364873 10930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "6abebc6c70774aca9ae022449650ca25"
format_stamp: "Formatted at 2026-09-26 13:40:49 on dist-test-slave-1wns"
server_key: "4e6a41a54960634ab205a899077d0fda"
server_key_iv: "2d2224bace52f5cc3b6ce3ee8e93ebc8"
server_key_version: "encryptionkey@0"
I20260926 13:40:49.369227 10930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 13:40:49.372394 10944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:49.374058 10930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:49.374253 10930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "6abebc6c70774aca9ae022449650ca25"
format_stamp: "Formatted at 2026-09-26 13:40:49 on dist-test-slave-1wns"
server_key: "4e6a41a54960634ab205a899077d0fda"
server_key_iv: "2d2224bace52f5cc3b6ce3ee8e93ebc8"
server_key_version: "encryptionkey@0"
I20260926 13:40:49.374398 10930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:49.379352 10930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:49.380070 10930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:49.380265 10930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:49.387583 10930 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:34997
I20260926 13:40:49.387602 10996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:34997 every 8 connection(s)
I20260926 13:40:49.388723 10930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:49.392280 10997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:49.392352 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 10930
I20260926 13:40:49.392495 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:49.392798 1220 external_mini_cluster.cc:1428] Setting key 64406b8f634a4960982f82b32d5725f0
I20260926 13:40:49.399204 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:49.394267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.1.49.62:34997)
0926 13:40:49.395005 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:49.395025 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:49.395073 (+ 48us) server_negotiation.cc:407] Connection header received
0926 13:40:49.395947 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:49.395974 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:49.396469 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:49.396870 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:49.397614 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:49.397619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:49.397634 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:49.397645 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:49.397710 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:49.397733 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:49.397883 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:49.397956 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 13:40:49.398219 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":110,"threads_started":1}
I20260926 13:40:49.400156 10997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25: Bootstrap starting.
I20260926 13:40:49.402576 10997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:49.403384 10997 log.cc:824] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:49.405587 10997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25: No bootstrap required, opened a new log
I20260926 13:40:49.409317 10997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6abebc6c70774aca9ae022449650ca25" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34997 } }
I20260926 13:40:49.409584 10997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:49.409628 10997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6abebc6c70774aca9ae022449650ca25, State: Initialized, Role: FOLLOWER
I20260926 13:40:49.410099 10997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6abebc6c70774aca9ae022449650ca25" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34997 } }
I20260926 13:40:49.410228 10997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:49.410300 10997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:49.410441 10997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:49.411446 10997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6abebc6c70774aca9ae022449650ca25" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34997 } }
I20260926 13:40:49.411882 10997 leader_election.cc:302] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6abebc6c70774aca9ae022449650ca25; no voters:
I20260926 13:40:49.412233 10997 leader_election.cc:288] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:49.412392 11002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:49.412671 11002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [term 1 LEADER]: Becoming Leader. State: Replica: 6abebc6c70774aca9ae022449650ca25, State: Running, Role: LEADER
I20260926 13:40:49.412988 11002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6abebc6c70774aca9ae022449650ca25" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34997 } }
I20260926 13:40:49.413666 10997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:49.414690 11003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6abebc6c70774aca9ae022449650ca25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6abebc6c70774aca9ae022449650ca25" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34997 } } }
I20260926 13:40:49.414839 11003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:49.415170 11004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6abebc6c70774aca9ae022449650ca25. Latest consensus state: current_term: 1 leader_uuid: "6abebc6c70774aca9ae022449650ca25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6abebc6c70774aca9ae022449650ca25" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34997 } } }
I20260926 13:40:49.415251 11011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:49.415302 11004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25 [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:49.418793 11011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:49.425652 11011 catalog_manager.cc:1347] Generated new cluster ID: 40d6049b6c7e4076a2b52fa105f78ef0
I20260926 13:40:49.425760 11011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:49.457077 11011 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:49.457468 11011 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:49.464689 11011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6abebc6c70774aca9ae022449650ca25: Generated new TSK 0
I20260926 13:40:49.465775 11011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:40:49.480110 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:34997
--builtin_ntp_servers=127.1.49.20:46195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 13:40:49.604800 11021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:49.605118 11021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:49.605160 11021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:49.609560 11021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:49.609637 11021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:49.609699 11021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:40:49.615001 11021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:46195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:34997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:49.616305 11021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:49.617795 11021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:49.628779 11026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:49.628798 11027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:49.628759 11029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:49.629068 11021 server_base.cc:1034] running on GCE node
I20260926 13:40:49.629745 11021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:49.630524 11021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:49.631729 11021 hybrid_clock.cc:648] HybridClock initialized: now 1790430049631703 us; error 36 us; skew 500 ppm
I20260926 13:40:49.633864 11021 webserver.cc:460] Webserver started at http://127.1.49.1:34757/ using document root <none> and password file <none>
I20260926 13:40:49.634423 11021 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:49.634486 11021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:49.634661 11021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:49.636821 11021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "15cdcfe43ee74fbd9abc34aa8ab9cabf"
format_stamp: "Formatted at 2026-09-26 13:40:49 on dist-test-slave-1wns"
server_key: "2967e88f5e6d824491707886ac849c49"
server_key_iv: "d524aae37f7b91ed4e200137cd399f72"
server_key_version: "encryptionkey@0"
I20260926 13:40:49.637347 11021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "15cdcfe43ee74fbd9abc34aa8ab9cabf"
format_stamp: "Formatted at 2026-09-26 13:40:49 on dist-test-slave-1wns"
server_key: "2967e88f5e6d824491707886ac849c49"
server_key_iv: "d524aae37f7b91ed4e200137cd399f72"
server_key_version: "encryptionkey@0"
I20260926 13:40:49.641258 11021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 13:40:49.644070 11035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:49.645496 11021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:49.645668 11021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "15cdcfe43ee74fbd9abc34aa8ab9cabf"
format_stamp: "Formatted at 2026-09-26 13:40:49 on dist-test-slave-1wns"
server_key: "2967e88f5e6d824491707886ac849c49"
server_key_iv: "d524aae37f7b91ed4e200137cd399f72"
server_key_version: "encryptionkey@0"
I20260926 13:40:49.645807 11021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:49.650619 11021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:49.651301 11021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:49.651495 11021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:49.652137 11021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:49.653101 11021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:49.653172 11021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:49.653241 11021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:49.653291 11021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:49.662797 11021 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:34595
I20260926 13:40:49.662815 11148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:34595 every 8 connection(s)
I20260926 13:40:49.663970 11021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:40:49.667485 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 11021
I20260926 13:40:49.667624 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:40:49.668012 1220 external_mini_cluster.cc:1428] Setting key 034dc2a57447a86ebb5a52ac86aeb663
I20260926 13:40:49.670586 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:34997
--builtin_ntp_servers=127.1.49.20:46195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 13:40:49.671476 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:49.666024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:47575 (local address 127.1.49.62:34997)
0926 13:40:49.666169 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:49.666173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:49.667551 (+ 1378us) server_negotiation.cc:407] Connection header received
0926 13:40:49.669214 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:49.669217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:49.669256 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:49.669355 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:49.670973 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:49.670976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:49.670978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:49.670993 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:49.671016 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:49.671026 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:49.671114 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:49.671316 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 13:40:49.671360 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:40:49.672780 11151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:49.666517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:34997 (local address 127.1.49.1:47575)
0926 13:40:49.667320 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:49.667384 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:49.668981 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:49.669676 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:49.669691 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:49.669846 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:49.670534 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:40:49.670537 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:49.670602 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:40:49.670719 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:49.671131 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:49.671144 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:49.671155 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 13:40:49.671368 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 13:40:49.671662 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":125,"threads_started":1}
I20260926 13:40:49.674494 11149 heartbeater.cc:347] Connected to a master server at 127.1.49.62:34997
I20260926 13:40:49.674793 11149 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:49.675276 11149 heartbeater.cc:511] Master 127.1.49.62:34997 requested a full tablet report, sending...
I20260926 13:40:49.677260 10961 ts_manager.cc:194] Registered new tserver with Master: 15cdcfe43ee74fbd9abc34aa8ab9cabf (127.1.49.1:34595)
W20260926 13:40:49.790463 11152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:49.790757 11152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:49.790799 11152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:49.795092 11152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:49.795202 11152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:49.795331 11152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:40:49.801137 11152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:46195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:34997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:49.802367 11152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:49.803984 11152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:49.814733 11157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:49.814849 11152 server_base.cc:1034] running on GCE node
W20260926 13:40:49.814708 11160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:49.815126 11158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:49.815678 11152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:49.816524 11152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:49.817724 11152 hybrid_clock.cc:648] HybridClock initialized: now 1790430049817702 us; error 34 us; skew 500 ppm
I20260926 13:40:49.820014 11152 webserver.cc:460] Webserver started at http://127.1.49.2:39655/ using document root <none> and password file <none>
I20260926 13:40:49.820755 11152 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:49.820825 11152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:49.821012 11152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:49.823279 11152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "44de7570c3db47feb84ebb9bee21aa58"
format_stamp: "Formatted at 2026-09-26 13:40:49 on dist-test-slave-1wns"
server_key: "0312c30accb8beef45a569f0d926c398"
server_key_iv: "ef90458f28a57af0343c40bb0fc9f8d7"
server_key_version: "encryptionkey@0"
I20260926 13:40:49.823922 11152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "44de7570c3db47feb84ebb9bee21aa58"
format_stamp: "Formatted at 2026-09-26 13:40:49 on dist-test-slave-1wns"
server_key: "0312c30accb8beef45a569f0d926c398"
server_key_iv: "ef90458f28a57af0343c40bb0fc9f8d7"
server_key_version: "encryptionkey@0"
I20260926 13:40:49.828282 11152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 13:40:49.831216 11166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:49.832731 11152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:40:49.832929 11152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "44de7570c3db47feb84ebb9bee21aa58"
format_stamp: "Formatted at 2026-09-26 13:40:49 on dist-test-slave-1wns"
server_key: "0312c30accb8beef45a569f0d926c398"
server_key_iv: "ef90458f28a57af0343c40bb0fc9f8d7"
server_key_version: "encryptionkey@0"
I20260926 13:40:49.833063 11152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:49.837997 11152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:49.838678 11152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:49.838877 11152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:49.839466 11152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:49.840510 11152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:49.840559 11152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:49.840598 11152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:49.840662 11152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:49.850041 11152 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:44709
I20260926 13:40:49.850104 11279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:44709 every 8 connection(s)
I20260926 13:40:49.851104 11152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:40:49.858438 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:49.853230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:34935 (local address 127.1.49.62:34997)
0926 13:40:49.853422 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:49.853427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:49.854643 (+ 1216us) server_negotiation.cc:407] Connection header received
0926 13:40:49.856245 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:49.856248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:49.856292 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:49.856378 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:49.857760 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:49.857763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:49.857765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:49.857779 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:49.857800 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:49.857809 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:49.857903 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:49.858045 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 13:40:49.858250 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 13:40:49.858502 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 11152
I20260926 13:40:49.858634 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:40:49.858987 1220 external_mini_cluster.cc:1428] Setting key 2938e920e69294c56f8f43daf30ce9b2
I20260926 13:40:49.859382 11282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:49.853636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:34997 (local address 127.1.49.2:34935)
0926 13:40:49.854428 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:49.854478 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:49.855997 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:49.856604 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:49.856613 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:49.856719 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:49.857439 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:40:49.857441 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:49.857489 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:40:49.857579 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:49.857899 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:49.857906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:49.857911 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:40:49.858079 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 13:40:49.858307 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":125,"threads_started":1}
I20260926 13:40:49.860908 11280 heartbeater.cc:347] Connected to a master server at 127.1.49.62:34997
I20260926 13:40:49.861243 11280 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:49.861596 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:34997
--builtin_ntp_servers=127.1.49.20:46195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 13:40:49.861784 11280 heartbeater.cc:511] Master 127.1.49.62:34997 requested a full tablet report, sending...
I20260926 13:40:49.862968 10961 ts_manager.cc:194] Registered new tserver with Master: 44de7570c3db47feb84ebb9bee21aa58 (127.1.49.2:44709)
W20260926 13:40:49.979847 11283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:49.980165 11283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:49.980207 11283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:49.984431 11283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:49.984503 11283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:49.984591 11283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:40:49.989737 11283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:46195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:34997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:49.990958 11283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:49.992497 11283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:50.003116 11288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:50.003130 11291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:50.003142 11289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:50.003257 11283 server_base.cc:1034] running on GCE node
I20260926 13:40:50.003861 11283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:50.004709 11283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:50.005882 11283 hybrid_clock.cc:648] HybridClock initialized: now 1790430050005846 us; error 52 us; skew 500 ppm
I20260926 13:40:50.008699 11283 webserver.cc:460] Webserver started at http://127.1.49.3:44341/ using document root <none> and password file <none>
I20260926 13:40:50.009380 11283 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:50.009480 11283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:50.009707 11283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:50.012274 11283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "8318645af9834783a7a3880b17bf20b1"
format_stamp: "Formatted at 2026-09-26 13:40:50 on dist-test-slave-1wns"
server_key: "4d53d36d59a27c5a098e544917c5ab8e"
server_key_iv: "81480223c106842eb2ab78bced89021f"
server_key_version: "encryptionkey@0"
I20260926 13:40:50.012917 11283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "8318645af9834783a7a3880b17bf20b1"
format_stamp: "Formatted at 2026-09-26 13:40:50 on dist-test-slave-1wns"
server_key: "4d53d36d59a27c5a098e544917c5ab8e"
server_key_iv: "81480223c106842eb2ab78bced89021f"
server_key_version: "encryptionkey@0"
I20260926 13:40:50.017513 11283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:40:50.020916 11297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:50.022290 11283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 13:40:50.022435 11283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "8318645af9834783a7a3880b17bf20b1"
format_stamp: "Formatted at 2026-09-26 13:40:50 on dist-test-slave-1wns"
server_key: "4d53d36d59a27c5a098e544917c5ab8e"
server_key_iv: "81480223c106842eb2ab78bced89021f"
server_key_version: "encryptionkey@0"
I20260926 13:40:50.022578 11283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:50.027951 11283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:50.028753 11283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:50.028959 11283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:50.029578 11283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:50.030612 11283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:50.030694 11283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:50.030789 11283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:50.030840 11283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:50.040603 11283 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:44375
I20260926 13:40:50.040675 11410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:44375 every 8 connection(s)
I20260926 13:40:50.041735 11283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:40:50.049063 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 11283
I20260926 13:40:50.049192 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:40:50.049629 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:50.044091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:44537 (local address 127.1.49.62:34997)
0926 13:40:50.044272 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:50.044276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:50.045632 (+ 1356us) server_negotiation.cc:407] Connection header received
0926 13:40:50.047204 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:50.047208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:50.047257 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:50.047353 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:50.048825 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:50.048829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:50.048833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:40:50.048852 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:50.048882 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:40:50.048895 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:50.049055 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:50.049392 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 13:40:50.049482 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 13:40:50.049649 1220 external_mini_cluster.cc:1428] Setting key 6779f9477388567023a47e633def81a4
I20260926 13:40:50.050793 11413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:50.044602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:34997 (local address 127.1.49.3:44537)
0926 13:40:50.045427 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:50.045479 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:50.046857 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:50.047559 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:50.047567 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:50.047678 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:50.048452 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:40:50.048453 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:50.048502 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:40:50.048591 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:50.049052 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:50.049071 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:50.049077 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:40:50.049434 (+ 357us) client_negotiation.cc:240] Negotiation successful
0926 13:40:50.049747 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":129,"threads_started":1}
I20260926 13:40:50.052055 11411 heartbeater.cc:347] Connected to a master server at 127.1.49.62:34997
I20260926 13:40:50.052372 11411 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:50.052848 11411 heartbeater.cc:511] Master 127.1.49.62:34997 requested a full tablet report, sending...
I20260926 13:40:50.054051 10961 ts_manager.cc:194] Registered new tserver with Master: 8318645af9834783a7a3880b17bf20b1 (127.1.49.3:44375)
I20260926 13:40:50.064507 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 13:40:50.067791 11000 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:40:50.066691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.1.49.62:34997)
0926 13:40:50.066900 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:50.066904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:50.067124 (+ 220us) server_negotiation.cc:407] Connection header received
0926 13:40:50.067242 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:50.067244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:50.067279 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:50.067354 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:50.067572 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56890: BlockingRecv error: recv got EOF from 127.0.0.1:56890 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 13:40:50.068110 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 11021
I20260926 13:40:50.073992 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 11152
I20260926 13:40:50.079679 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 11283
I20260926 13:40:50.085582 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 10930
2026-09-26T13:40:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (910 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:40:50 dist-test-slave-1wns krb5kdc[11421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:40:50 dist-test-slave-1wns krb5kdc[11421](info): set up 2 sockets
Sep 26 13:40:50 dist-test-slave-1wns krb5kdc[11421](info): commencing operation
krb5kdc: starting...
W20260926 13:40:51.128922 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:40:51 dist-test-slave-1wns krb5kdc[11421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:40:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:40:51.267908 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:41611
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:41611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:51.393136 11434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:51.393491 11434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:51.393537 11434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:51.398396 11434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:40:51.398486 11434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:51.398506 11434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:51.398527 11434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:40:51.398543 11434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:40:51.404266 11434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:41611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:41611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:51.405464 11434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:51.406508 11434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:51.415683 11440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:51.415676 11442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:51.415683 11439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:51.416034 11434 server_base.cc:1034] running on GCE node
I20260926 13:40:51.416836 11434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:51.418017 11434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:51.419269 11434 hybrid_clock.cc:648] HybridClock initialized: now 1790430051419256 us; error 30 us; skew 500 ppm
Sep 26 13:40:51 dist-test-slave-1wns krb5kdc[11421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430051, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:51.422251 11434 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:40:51.423198 11434 webserver.cc:460] Webserver started at http://127.1.49.62:33909/ using document root <none> and password file <none>
I20260926 13:40:51.423849 11434 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:51.423950 11434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:51.424233 11434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:51.426481 11434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "5467144dba894a309b61c39b67fc984c"
format_stamp: "Formatted at 2026-09-26 13:40:51 on dist-test-slave-1wns"
server_key: "20f153dfe55d92c1cd776992634119d2"
server_key_iv: "c222e575877f56be26825ff6a614f034"
server_key_version: "encryptionkey@0"
I20260926 13:40:51.427038 11434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "5467144dba894a309b61c39b67fc984c"
format_stamp: "Formatted at 2026-09-26 13:40:51 on dist-test-slave-1wns"
server_key: "20f153dfe55d92c1cd776992634119d2"
server_key_iv: "c222e575877f56be26825ff6a614f034"
server_key_version: "encryptionkey@0"
I20260926 13:40:51.431136 11434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 13:40:51.433826 11449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:51.435348 11434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 13:40:51.435463 11434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "5467144dba894a309b61c39b67fc984c"
format_stamp: "Formatted at 2026-09-26 13:40:51 on dist-test-slave-1wns"
server_key: "20f153dfe55d92c1cd776992634119d2"
server_key_iv: "c222e575877f56be26825ff6a614f034"
server_key_version: "encryptionkey@0"
I20260926 13:40:51.435571 11434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:51.455955 11434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:51.459728 11434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:51.460009 11434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:51.468043 11434 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:41611
I20260926 13:40:51.468367 11501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:41611 every 8 connection(s)
I20260926 13:40:51.469599 11434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:40:51.472970 11502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:51.475716 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 11434
I20260926 13:40:51.475875 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:40:51.476203 1220 external_mini_cluster.cc:1428] Setting key 0adb79f5cf77b8ebe75d43b8496b33f8
I20260926 13:40:51.480023 11502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c: Bootstrap starting.
I20260926 13:40:51.483062 11502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:51.484030 11502 log.cc:824] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:51.486058 11502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c: No bootstrap required, opened a new log
Sep 26 13:40:51 dist-test-slave-1wns krb5kdc[11421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:51.490062 11502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5467144dba894a309b61c39b67fc984c" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41611 } }
I20260926 13:40:51.490341 11502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:51.490429 11502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5467144dba894a309b61c39b67fc984c, State: Initialized, Role: FOLLOWER
I20260926 13:40:51.490943 11502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5467144dba894a309b61c39b67fc984c" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41611 } }
I20260926 13:40:51.491108 11502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:40:51.491197 11502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:40:51.491322 11502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:51.493032 11502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5467144dba894a309b61c39b67fc984c" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41611 } }
I20260926 13:40:51.493346 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:51.477863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.1.49.62:41611)
0926 13:40:51.478324 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:51.478345 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:51.478393 (+ 48us) server_negotiation.cc:407] Connection header received
0926 13:40:51.479290 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:51.479317 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:51.479772 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:51.480231 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:51.481004 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:51.482670 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:51.483670 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:51.484481 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:51.484689 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:51.487498 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:51.487504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:51.487520 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:51.487548 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:51.489567 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:51.490119 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:51.490192 (+ 73us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:51.490199 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:51.490328 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:51.490743 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:51.490747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:51.490749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:51.491369 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:51.491574 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:51.491950 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 13:40:51.492363 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":121,"threads_started":1}
I20260926 13:40:51.493557 11502 leader_election.cc:302] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5467144dba894a309b61c39b67fc984c; no voters:
I20260926 13:40:51.493958 11502 leader_election.cc:288] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:40:51.494097 11507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:51.494325 11507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [term 1 LEADER]: Becoming Leader. State: Replica: 5467144dba894a309b61c39b67fc984c, State: Running, Role: LEADER
I20260926 13:40:51.494745 11507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5467144dba894a309b61c39b67fc984c" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41611 } }
I20260926 13:40:51.495290 11502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:40:51.496590 11508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5467144dba894a309b61c39b67fc984c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5467144dba894a309b61c39b67fc984c" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41611 } } }
I20260926 13:40:51.496742 11508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:51.497185 11516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:40:51.497166 11509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5467144dba894a309b61c39b67fc984c. Latest consensus state: current_term: 1 leader_uuid: "5467144dba894a309b61c39b67fc984c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5467144dba894a309b61c39b67fc984c" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41611 } } }
I20260926 13:40:51.497383 11509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c [sys.catalog]: This master's current role is: LEADER
I20260926 13:40:51.500499 11516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:40:51.507319 11516 catalog_manager.cc:1347] Generated new cluster ID: 366eecac0f334a1a87f0d5cc05832b56
I20260926 13:40:51.507421 11516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:40:51.517664 11516 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:40:51.518996 11516 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:40:51.535880 11516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c: Generated new TSK 0
I20260926 13:40:51.536830 11516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:40:51.575291 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41611
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:51.695302 11528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:51.695619 11528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:51.695665 11528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:51.700232 11528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:51.700320 11528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:51.700384 11528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:40:51.706265 11528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:51.707561 11528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:51.708799 11528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:51.719776 11533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:51.719879 11534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:51.719852 11536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:51.719822 11528 server_base.cc:1034] running on GCE node
I20260926 13:40:51.720476 11528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:51.721346 11528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:51.722564 11528 hybrid_clock.cc:648] HybridClock initialized: now 1790430051722544 us; error 32 us; skew 500 ppm
Sep 26 13:40:51 dist-test-slave-1wns krb5kdc[11421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430051, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:51.725664 11528 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:40:51.726697 11528 webserver.cc:460] Webserver started at http://127.1.49.1:40367/ using document root <none> and password file <none>
I20260926 13:40:51.727325 11528 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:51.727418 11528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:51.727665 11528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:51.729944 11528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "9b394f4e1e314d2db2a6e736a9c79249"
format_stamp: "Formatted at 2026-09-26 13:40:51 on dist-test-slave-1wns"
server_key: "36977db4681300b62ab1c88c7a30f1e7"
server_key_iv: "105f879c67b787fdcab098b4abd7da63"
server_key_version: "encryptionkey@0"
I20260926 13:40:51.730527 11528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "9b394f4e1e314d2db2a6e736a9c79249"
format_stamp: "Formatted at 2026-09-26 13:40:51 on dist-test-slave-1wns"
server_key: "36977db4681300b62ab1c88c7a30f1e7"
server_key_iv: "105f879c67b787fdcab098b4abd7da63"
server_key_version: "encryptionkey@0"
I20260926 13:40:51.734639 11528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 13:40:51.737639 11543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:51.739220 11528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:40:51.739384 11528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "9b394f4e1e314d2db2a6e736a9c79249"
format_stamp: "Formatted at 2026-09-26 13:40:51 on dist-test-slave-1wns"
server_key: "36977db4681300b62ab1c88c7a30f1e7"
server_key_iv: "105f879c67b787fdcab098b4abd7da63"
server_key_version: "encryptionkey@0"
I20260926 13:40:51.739534 11528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:51.760756 11528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:51.764300 11528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:51.764524 11528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:51.765169 11528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:51.766067 11528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:51.766196 11528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:51.766274 11528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:51.766311 11528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:51.776427 11528 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:35323
I20260926 13:40:51.776454 11656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:35323 every 8 connection(s)
I20260926 13:40:51.777434 11528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:40:51.783347 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 11528
I20260926 13:40:51.783490 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:40:51.783864 1220 external_mini_cluster.cc:1428] Setting key 1cbd579e42392a9c009be2a6501adbcd
Sep 26 13:40:51 dist-test-slave-1wns krb5kdc[11421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430051, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
I20260926 13:40:51.794994 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:51.779471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:39089 (local address 127.1.49.62:41611)
0926 13:40:51.779683 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:51.779687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:51.780739 (+ 1052us) server_negotiation.cc:407] Connection header received
0926 13:40:51.782213 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:51.782216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:51.782262 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:51.782339 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:51.784150 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:51.784846 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:51.786655 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:51.786929 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:51.787043 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:51.790496 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:51.790501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:51.790506 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:51.790538 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:51.792489 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:51.793135 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:51.793140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:51.793142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:51.793251 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:51.793681 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:51.793685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:51.793687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:51.793945 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:51.794060 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:51.794632 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 13:40:51.794763 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 13:40:51.796090 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:51.779995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41611 (local address 127.1.49.1:39089)
0926 13:40:51.780565 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:51.780615 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:51.781969 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:51.782535 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:51.782542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:51.783106 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:51.783941 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:51.783966 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:51.784983 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:51.784987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:51.786460 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:51.786476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:51.787131 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:51.787139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:51.787428 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:51.788288 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:51.788307 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:51.790265 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:51.792672 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:51.792679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:51.792696 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:51.792996 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:51.793379 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:51.793381 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:51.793383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:51.793559 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:51.794075 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:51.794081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:51.794354 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 13:40:51.794520 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 13:40:51.794851 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":162,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
I20260926 13:40:51.800805 11657 heartbeater.cc:347] Connected to a master server at 127.1.49.62:41611
I20260926 13:40:51.801154 11657 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:51.801775 11657 heartbeater.cc:511] Master 127.1.49.62:41611 requested a full tablet report, sending...
I20260926 13:40:51.803637 11466 ts_manager.cc:194] Registered new tserver with Master: 9b394f4e1e314d2db2a6e736a9c79249 (127.1.49.1:35323)
I20260926 13:40:51.804908 11466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:39089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:40:51.817464 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41611
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:51.941331 11662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:51.941661 11662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:51.941740 11662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:51.946336 11662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:51.946448 11662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:51.946544 11662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:40:51.952366 11662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:51.953634 11662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:51.954783 11662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:40:51.965901 11670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:51.965910 11667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:51.965965 11668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:51.966300 11662 server_base.cc:1034] running on GCE node
I20260926 13:40:51.966815 11662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:51.967662 11662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:51.968946 11662 hybrid_clock.cc:648] HybridClock initialized: now 1790430051968923 us; error 43 us; skew 500 ppm
Sep 26 13:40:51 dist-test-slave-1wns krb5kdc[11421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430051, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:51.979046 11662 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:40:51.980309 11662 webserver.cc:460] Webserver started at http://127.1.49.2:35367/ using document root <none> and password file <none>
I20260926 13:40:51.981009 11662 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:51.981123 11662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:51.981397 11662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:51.983855 11662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "d6032751fcde490e92bedb2adb035ae2"
format_stamp: "Formatted at 2026-09-26 13:40:51 on dist-test-slave-1wns"
server_key: "1e439ba2dc2c4d769047e769cde748d2"
server_key_iv: "d13659004f8c9d24daa438f3a7a96fb0"
server_key_version: "encryptionkey@0"
I20260926 13:40:51.984544 11662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "d6032751fcde490e92bedb2adb035ae2"
format_stamp: "Formatted at 2026-09-26 13:40:51 on dist-test-slave-1wns"
server_key: "1e439ba2dc2c4d769047e769cde748d2"
server_key_iv: "d13659004f8c9d24daa438f3a7a96fb0"
server_key_version: "encryptionkey@0"
I20260926 13:40:51.989276 11662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:40:51.992317 11677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:51.993842 11662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:40:51.994000 11662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "d6032751fcde490e92bedb2adb035ae2"
format_stamp: "Formatted at 2026-09-26 13:40:51 on dist-test-slave-1wns"
server_key: "1e439ba2dc2c4d769047e769cde748d2"
server_key_iv: "d13659004f8c9d24daa438f3a7a96fb0"
server_key_version: "encryptionkey@0"
I20260926 13:40:51.994135 11662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:52.010609 11662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:52.014742 11662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:52.014983 11662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:52.015864 11662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:52.016870 11662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:52.016933 11662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:52.017014 11662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:52.017055 11662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:52.026583 11662 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:34585
I20260926 13:40:52.026605 11790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:34585 every 8 connection(s)
I20260926 13:40:52.027660 11662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:40:52.035352 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 11662
I20260926 13:40:52.035462 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:40:52.035789 1220 external_mini_cluster.cc:1428] Setting key 3469b188f606675cba6dcd43e7cd62f8
Sep 26 13:40:52 dist-test-slave-1wns krb5kdc[11421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430051, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:52.046018 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.030038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:34341 (local address 127.1.49.62:41611)
0926 13:40:52.030195 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:52.030199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:52.031201 (+ 1002us) server_negotiation.cc:407] Connection header received
0926 13:40:52.032784 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:52.032788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:52.032846 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:52.032945 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:52.034475 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.035067 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.036998 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.037487 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.037647 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.041197 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:52.041202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:52.041204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:52.041239 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:52.043380 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:52.044293 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:52.044302 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:52.044303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:52.044391 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:52.044789 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:52.044793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:52.044794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:52.045008 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:52.045123 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:52.045708 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 13:40:52.045849 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
I20260926 13:40:52.047021 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.030464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41611 (local address 127.1.49.2:34341)
0926 13:40:52.031018 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:52.031069 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:52.032492 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:52.033282 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:52.033291 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:52.033730 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:52.034282 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:52.034297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.035290 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.035293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:52.036832 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:52.036845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.037848 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.037851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:52.038019 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.038804 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:52.038824 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:52.040930 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:52.043549 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:52.043557 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:52.043584 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:52.044069 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:52.044494 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:52.044497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:52.044499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:52.044680 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:52.045124 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:52.045133 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:52.045436 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 13:40:52.045646 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 13:40:52.046045 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":135,"threads_started":1}
I20260926 13:40:52.048542 11791 heartbeater.cc:347] Connected to a master server at 127.1.49.62:41611
I20260926 13:40:52.048911 11791 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:52.049571 11791 heartbeater.cc:511] Master 127.1.49.62:41611 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
I20260926 13:40:52.050920 11466 ts_manager.cc:194] Registered new tserver with Master: d6032751fcde490e92bedb2adb035ae2 (127.1.49.2:34585)
I20260926 13:40:52.051793 11466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:34341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:40:52.071120 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41611
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:45167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:40:52.198613 11796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:40:52.199028 11796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:40:52.199103 11796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:40:52.203490 11796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:40:52.203585 11796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:40:52.203651 11796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:40:52.209534 11796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:45167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:40:52.210798 11796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:40:52.212095 11796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:40:52.223125 11796 server_base.cc:1034] running on GCE node
W20260926 13:40:52.223115 11801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:52.223112 11802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:40:52.223414 11804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:40:52.224020 11796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:40:52.224774 11796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:40:52.225989 11796 hybrid_clock.cc:648] HybridClock initialized: now 1790430052225959 us; error 44 us; skew 500 ppm
Sep 26 13:40:52 dist-test-slave-1wns krb5kdc[11421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430052, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:40:52.229555 11796 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:40:52.230542 11796 webserver.cc:460] Webserver started at http://127.1.49.3:33437/ using document root <none> and password file <none>
I20260926 13:40:52.231124 11796 fs_manager.cc:359] Metadata directory not provided
I20260926 13:40:52.231191 11796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:40:52.231371 11796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:40:52.233672 11796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "6caa3d004fb14841a119d40670acbb1b"
format_stamp: "Formatted at 2026-09-26 13:40:52 on dist-test-slave-1wns"
server_key: "b6b259861f69de6044a2d0e836269487"
server_key_iv: "7f91aefe80e2f24fdc2370491400d0d1"
server_key_version: "encryptionkey@0"
I20260926 13:40:52.234251 11796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "6caa3d004fb14841a119d40670acbb1b"
format_stamp: "Formatted at 2026-09-26 13:40:52 on dist-test-slave-1wns"
server_key: "b6b259861f69de6044a2d0e836269487"
server_key_iv: "7f91aefe80e2f24fdc2370491400d0d1"
server_key_version: "encryptionkey@0"
I20260926 13:40:52.238483 11796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:40:52.241412 11811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:52.243002 11796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:40:52.243127 11796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "6caa3d004fb14841a119d40670acbb1b"
format_stamp: "Formatted at 2026-09-26 13:40:52 on dist-test-slave-1wns"
server_key: "b6b259861f69de6044a2d0e836269487"
server_key_iv: "7f91aefe80e2f24fdc2370491400d0d1"
server_key_version: "encryptionkey@0"
I20260926 13:40:52.243252 11796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:40:52.260183 11796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:40:52.263556 11796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:40:52.263739 11796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:40:52.264412 11796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:40:52.265270 11796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:40:52.265321 11796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:52.265364 11796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:40:52.265379 11796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:40:52.274734 11796 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:38065
I20260926 13:40:52.274808 11924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:38065 every 8 connection(s)
I20260926 13:40:52.275838 11796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:40:52.279435 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 11796
I20260926 13:40:52.279579 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:40:52.279953 1220 external_mini_cluster.cc:1428] Setting key 9c9873ac3543f44a6e88fac21c0cbead
Sep 26 13:40:52 dist-test-slave-1wns krb5kdc[11421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430052, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:40:52.291666 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.278159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:56571 (local address 127.1.49.62:41611)
0926 13:40:52.278336 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:52.278341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:52.279505 (+ 1164us) server_negotiation.cc:407] Connection header received
0926 13:40:52.281254 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:52.281258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:52.281320 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:52.281426 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:52.282852 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.283388 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.284614 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.284845 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.284961 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.287722 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:52.287725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:52.287728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:52.287759 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:52.289366 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:52.289957 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:52.289961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:52.289963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:52.290054 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:52.290434 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:52.290438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:52.290439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:52.290649 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:52.290761 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:52.291318 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 13:40:52.291481 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 13:40:52.293203 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.278745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41611 (local address 127.1.49.3:56571)
0926 13:40:52.279311 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:52.279367 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:52.280958 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:52.281632 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:52.281640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:52.282156 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:40:52.282695 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:52.282710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.283520 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.283526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:52.284478 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:52.284487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.285007 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.285011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:52.285202 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.285890 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:40:52.285907 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:40:52.287506 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:40:52.289505 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:52.289513 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:52.289533 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:52.289838 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:52.290151 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:40:52.290154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:40:52.290158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:40:52.290322 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:40:52.290759 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:40:52.290768 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:40:52.291049 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 13:40:52.291284 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 13:40:52.291954 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":137,"threads_started":1}
I20260926 13:40:52.294440 11925 heartbeater.cc:347] Connected to a master server at 127.1.49.62:41611
I20260926 13:40:52.294693 11925 heartbeater.cc:464] Registering TS with master...
I20260926 13:40:52.295212 11925 heartbeater.cc:511] Master 127.1.49.62:41611 requested a full tablet report, sending...
I20260926 13:40:52.296527 11465 ts_manager.cc:194] Registered new tserver with Master: 6caa3d004fb14841a119d40670acbb1b (127.1.49.3:38065)
I20260926 13:40:52.297219 11465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:56571
I20260926 13:40:52.306658 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:40:52.318061 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.309080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.1.49.62:41611)
0926 13:40:52.309238 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:52.309242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:52.309351 (+ 109us) server_negotiation.cc:407] Connection header received
0926 13:40:52.309447 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:52.309450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:52.309497 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:52.309567 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:40:52.310547 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.311167 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.312173 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.312415 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.312520 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.314190 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:40:52.314194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:40:52.314196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:40:52.314226 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:40:52.316017 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:52.316629 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:52.316636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:52.316640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:52.316751 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:40:52.317060 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:40:52.317064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:40:52.317065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:40:52.317334 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:40:52.317454 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:52.317808 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 13:40:52.317928 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 13:40:52.321971 11465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:40:52.325387 11465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:40:52.341892 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.336161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.1.49.3:38065)
0926 13:40:52.336649 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:52.336661 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:52.336684 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:40:52.336763 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:52.336769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:52.336990 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:52.337168 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:52.337355 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.338858 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.340525 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.341105 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.341242 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.341414 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:52.341607 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 13:40:52.341738 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":143,"threads_started":1}
I20260926 13:40:52.341879 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.335443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:38065 (local address 127.0.0.1:48606)
0926 13:40:52.336310 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:52.336326 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:52.336412 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:52.337134 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:52.337137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:52.337156 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:52.337263 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:52.337269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.339015 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.339018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:52.340346 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:52.340358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.341244 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.341247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:52.341379 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.341417 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 13:40:52.341553 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 13:40:52.341706 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":107,"threads_started":1}
I20260926 13:40:52.342239 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.335523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44902 (local address 127.1.49.2:34585)
0926 13:40:52.336151 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:52.336166 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:52.336189 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:40:52.336284 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:52.336289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:52.336519 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:52.336687 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:52.337263 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.338327 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.340293 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.341163 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.341307 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.341451 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:52.341951 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 13:40:52.342090 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":124,"threads_started":1}
I20260926 13:40:52.342280 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.336698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37586 (local address 127.1.49.1:35323)
0926 13:40:52.337106 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:52.337113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:52.337135 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:40:52.337208 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:52.337214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:52.337409 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:52.337690 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:52.337948 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.339098 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.340480 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.341230 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.341328 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.341451 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:52.342062 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 13:40:52.342173 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":111,"threads_started":1}
I20260926 13:40:52.341879 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.335213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:34585 (local address 127.0.0.1:44902)
0926 13:40:52.335682 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:52.335738 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:52.335911 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:52.336922 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:52.336930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:52.336956 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:52.337118 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:52.337126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.338453 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.338459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:52.340131 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:52.340140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.341493 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.341500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:52.341597 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.341617 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 13:40:52.341682 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:40:52.341732 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":84,"threads_started":1}
I20260926 13:40:52.341954 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.335980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:35323 (local address 127.0.0.1:37586)
0926 13:40:52.336880 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:52.336893 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:52.336971 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:52.337661 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:52.337664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:52.337683 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:52.337799 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:52.337805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.339221 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.339224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:52.340359 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:52.340365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.341521 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.341526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:52.341630 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.341654 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 13:40:52.341737 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 13:40:52.341812 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"spinlock_wait_cycles":1408,"thread_start_us":95,"threads_started":1}
I20260926 13:40:52.345932 11725 tablet_service.cc:1467] Processing CreateTablet for tablet 6414ea9130db4a37b363f2d08b661ed8 (DEFAULT_TABLE table=test-table [id=009b883803b14ce3a32927d5524e4a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:52.345937 11591 tablet_service.cc:1467] Processing CreateTablet for tablet 6414ea9130db4a37b363f2d08b661ed8 (DEFAULT_TABLE table=test-table [id=009b883803b14ce3a32927d5524e4a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:52.345937 11859 tablet_service.cc:1467] Processing CreateTablet for tablet 6414ea9130db4a37b363f2d08b661ed8 (DEFAULT_TABLE table=test-table [id=009b883803b14ce3a32927d5524e4a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:40:52.347149 11591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6414ea9130db4a37b363f2d08b661ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:52.347152 11859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6414ea9130db4a37b363f2d08b661ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:52.347148 11725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6414ea9130db4a37b363f2d08b661ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:40:52.354075 11942 tablet_bootstrap.cc:492] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2: Bootstrap starting.
I20260926 13:40:52.355432 11943 tablet_bootstrap.cc:492] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249: Bootstrap starting.
I20260926 13:40:52.355432 11944 tablet_bootstrap.cc:492] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b: Bootstrap starting.
I20260926 13:40:52.357159 11942 tablet_bootstrap.cc:654] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:52.357829 11944 tablet_bootstrap.cc:654] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:52.357893 11943 tablet_bootstrap.cc:654] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249: Neither blocks nor log segments found. Creating new log.
I20260926 13:40:52.358180 11942 log.cc:824] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:52.358834 11943 log.cc:824] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:52.359164 11944 log.cc:824] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:40:52.360675 11942 tablet_bootstrap.cc:492] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2: No bootstrap required, opened a new log
I20260926 13:40:52.360888 11943 tablet_bootstrap.cc:492] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249: No bootstrap required, opened a new log
I20260926 13:40:52.360913 11942 ts_tablet_manager.cc:1397] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260926 13:40:52.361101 11943 ts_tablet_manager.cc:1397] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:40:52.361255 11944 tablet_bootstrap.cc:492] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b: No bootstrap required, opened a new log
I20260926 13:40:52.361438 11944 ts_tablet_manager.cc:1397] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:40:52.364871 11943 raft_consensus.cc:348] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6032751fcde490e92bedb2adb035ae2" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34585 } } peers { permanent_uuid: "6caa3d004fb14841a119d40670acbb1b" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 38065 } } peers { permanent_uuid: "9b394f4e1e314d2db2a6e736a9c79249" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35323 } }
I20260926 13:40:52.365185 11943 raft_consensus.cc:374] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:52.365245 11943 raft_consensus.cc:729] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b394f4e1e314d2db2a6e736a9c79249, State: Initialized, Role: FOLLOWER
I20260926 13:40:52.365196 11942 raft_consensus.cc:348] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6032751fcde490e92bedb2adb035ae2" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34585 } } peers { permanent_uuid: "6caa3d004fb14841a119d40670acbb1b" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 38065 } } peers { permanent_uuid: "9b394f4e1e314d2db2a6e736a9c79249" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35323 } }
I20260926 13:40:52.365299 11944 raft_consensus.cc:348] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6032751fcde490e92bedb2adb035ae2" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34585 } } peers { permanent_uuid: "6caa3d004fb14841a119d40670acbb1b" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 38065 } } peers { permanent_uuid: "9b394f4e1e314d2db2a6e736a9c79249" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35323 } }
I20260926 13:40:52.365509 11944 raft_consensus.cc:374] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:52.365547 11942 raft_consensus.cc:374] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:40:52.365558 11944 raft_consensus.cc:729] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6caa3d004fb14841a119d40670acbb1b, State: Initialized, Role: FOLLOWER
I20260926 13:40:52.365607 11942 raft_consensus.cc:729] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6032751fcde490e92bedb2adb035ae2, State: Initialized, Role: FOLLOWER
I20260926 13:40:52.365751 11943 consensus_queue.cc:260] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6032751fcde490e92bedb2adb035ae2" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34585 } } peers { permanent_uuid: "6caa3d004fb14841a119d40670acbb1b" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 38065 } } peers { permanent_uuid: "9b394f4e1e314d2db2a6e736a9c79249" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35323 } }
I20260926 13:40:52.366047 11942 consensus_queue.cc:260] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6032751fcde490e92bedb2adb035ae2" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34585 } } peers { permanent_uuid: "6caa3d004fb14841a119d40670acbb1b" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 38065 } } peers { permanent_uuid: "9b394f4e1e314d2db2a6e736a9c79249" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35323 } }
I20260926 13:40:52.366047 11944 consensus_queue.cc:260] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6032751fcde490e92bedb2adb035ae2" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34585 } } peers { permanent_uuid: "6caa3d004fb14841a119d40670acbb1b" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 38065 } } peers { permanent_uuid: "9b394f4e1e314d2db2a6e736a9c79249" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35323 } }
I20260926 13:40:52.366999 11943 ts_tablet_manager.cc:1428] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260926 13:40:52.367069 11944 ts_tablet_manager.cc:1428] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:40:52.367069 11942 ts_tablet_manager.cc:1428] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260926 13:40:52.367079 11657 heartbeater.cc:503] Master 127.1.49.62:41611 was elected leader, sending a full tablet report...
I20260926 13:40:52.367395 11791 heartbeater.cc:503] Master 127.1.49.62:41611 was elected leader, sending a full tablet report...
I20260926 13:40:52.367395 11925 heartbeater.cc:503] Master 127.1.49.62:41611 was elected leader, sending a full tablet report...
I20260926 13:40:52.480224 11950 raft_consensus.cc:482] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:40:52.480504 11950 raft_consensus.cc:504] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6032751fcde490e92bedb2adb035ae2" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34585 } } peers { permanent_uuid: "6caa3d004fb14841a119d40670acbb1b" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 38065 } } peers { permanent_uuid: "9b394f4e1e314d2db2a6e736a9c79249" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35323 } }
I20260926 13:40:52.481738 11950 leader_election.cc:288] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6caa3d004fb14841a119d40670acbb1b (127.1.49.3:38065), 9b394f4e1e314d2db2a6e736a9c79249 (127.1.49.1:35323)
I20260926 13:40:52.487053 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.482267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:37101 (local address 127.1.49.3:38065)
0926 13:40:52.482404 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:52.482408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:52.482871 (+ 463us) server_negotiation.cc:407] Connection header received
0926 13:40:52.482933 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:52.482935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:52.482981 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:52.483151 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:52.483410 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.484011 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.485657 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.486281 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.486403 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.486470 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:52.486841 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 13:40:52.486919 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 13:40:52.487053 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.482162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:40727 (local address 127.1.49.1:35323)
0926 13:40:52.482295 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:52.482302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:52.482321 (+ 19us) server_negotiation.cc:407] Connection header received
0926 13:40:52.482377 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:52.482381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:52.482434 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:52.482538 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:40:52.483260 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.483893 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.485642 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.486301 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.486403 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.486470 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:52.486841 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 13:40:52.486922 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:40:52.487116 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.482040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:38065 (local address 127.1.49.2:37101)
0926 13:40:52.482746 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:52.482760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:52.482860 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:52.483178 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:52.483180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:52.483196 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:52.483284 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:52.483288 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.484138 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.484141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:52.485474 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:52.485479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.486431 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.486435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:52.486636 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.486704 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 13:40:52.486798 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 13:40:52.486933 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":434,"thread_start_us":179,"threads_started":1}
I20260926 13:40:52.487123 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.482040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:35323 (local address 127.1.49.2:40727)
0926 13:40:52.482153 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:52.482171 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:40:52.482266 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:40:52.482579 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:40:52.482584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:40:52.482610 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:40:52.482732 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:52.482738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.484009 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.484012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:52.485465 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:40:52.485477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.486431 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.486435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:40:52.486651 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.486702 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 13:40:52.486798 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 13:40:52.486933 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":188}
I20260926 13:40:52.488302 11879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6414ea9130db4a37b363f2d08b661ed8" candidate_uuid: "d6032751fcde490e92bedb2adb035ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6caa3d004fb14841a119d40670acbb1b" is_pre_election: true
I20260926 13:40:52.488302 11611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6414ea9130db4a37b363f2d08b661ed8" candidate_uuid: "d6032751fcde490e92bedb2adb035ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b394f4e1e314d2db2a6e736a9c79249" is_pre_election: true
I20260926 13:40:52.488646 11611 raft_consensus.cc:2454] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6032751fcde490e92bedb2adb035ae2 in term 0.
I20260926 13:40:52.488664 11879 raft_consensus.cc:2454] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6032751fcde490e92bedb2adb035ae2 in term 0.
I20260926 13:40:52.489215 11680 leader_election.cc:302] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6caa3d004fb14841a119d40670acbb1b, 9b394f4e1e314d2db2a6e736a9c79249, d6032751fcde490e92bedb2adb035ae2; no voters:
I20260926 13:40:52.489550 11950 raft_consensus.cc:2781] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:40:52.489647 11950 raft_consensus.cc:482] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:40:52.489691 11950 raft_consensus.cc:3037] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:52.490793 11950 raft_consensus.cc:504] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6032751fcde490e92bedb2adb035ae2" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34585 } } peers { permanent_uuid: "6caa3d004fb14841a119d40670acbb1b" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 38065 } } peers { permanent_uuid: "9b394f4e1e314d2db2a6e736a9c79249" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35323 } }
I20260926 13:40:52.491228 11950 leader_election.cc:288] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [CANDIDATE]: Term 1 election: Requested vote from peers 6caa3d004fb14841a119d40670acbb1b (127.1.49.3:38065), 9b394f4e1e314d2db2a6e736a9c79249 (127.1.49.1:35323)
I20260926 13:40:52.491882 11879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6414ea9130db4a37b363f2d08b661ed8" candidate_uuid: "d6032751fcde490e92bedb2adb035ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6caa3d004fb14841a119d40670acbb1b"
I20260926 13:40:52.491894 11611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6414ea9130db4a37b363f2d08b661ed8" candidate_uuid: "d6032751fcde490e92bedb2adb035ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b394f4e1e314d2db2a6e736a9c79249"
I20260926 13:40:52.492036 11879 raft_consensus.cc:3037] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:52.492182 11611 raft_consensus.cc:3037] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:40:52.493705 11879 raft_consensus.cc:2454] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6032751fcde490e92bedb2adb035ae2 in term 1.
I20260926 13:40:52.493968 11611 raft_consensus.cc:2454] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6032751fcde490e92bedb2adb035ae2 in term 1.
I20260926 13:40:52.494112 11680 leader_election.cc:302] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6caa3d004fb14841a119d40670acbb1b, d6032751fcde490e92bedb2adb035ae2; no voters:
I20260926 13:40:52.494342 11950 raft_consensus.cc:2781] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:40:52.494565 11950 raft_consensus.cc:686] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [term 1 LEADER]: Becoming Leader. State: Replica: d6032751fcde490e92bedb2adb035ae2, State: Running, Role: LEADER
I20260926 13:40:52.494984 11950 consensus_queue.cc:237] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6032751fcde490e92bedb2adb035ae2" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34585 } } peers { permanent_uuid: "6caa3d004fb14841a119d40670acbb1b" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 38065 } } peers { permanent_uuid: "9b394f4e1e314d2db2a6e736a9c79249" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35323 } }
I20260926 13:40:52.499411 11465 catalog_manager.cc:5495] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6032751fcde490e92bedb2adb035ae2 (127.1.49.2). New cstate: current_term: 1 leader_uuid: "d6032751fcde490e92bedb2adb035ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6032751fcde490e92bedb2adb035ae2" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 34585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6caa3d004fb14841a119d40670acbb1b" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 38065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b394f4e1e314d2db2a6e736a9c79249" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35323 } health_report { overall_health: UNKNOWN } } }
W20260926 13:40:52.527225 11926 tablet.cc:2369] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:40:52.528769 11658 tablet.cc:2369] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:40:52.529006 11792 tablet.cc:2369] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:40:52.535283 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.531593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.1.49.2:34585)
0926 13:40:52.531740 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 13:40:52.531746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:40:52.531762 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:40:52.531874 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:40:52.531876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:40:52.531933 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:40:52.532056 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:40:52.532361 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.532885 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.533781 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:40:52.533997 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:40:52.534106 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:40:52.534421 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:40:52.534861 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:40:52.534996 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 13:40:52.535058 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 13:40:52.535118 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:40:52.541759 11611 raft_consensus.cc:1261] T 6414ea9130db4a37b363f2d08b661ed8 P 9b394f4e1e314d2db2a6e736a9c79249 [term 1 FOLLOWER]: Refusing update from remote peer d6032751fcde490e92bedb2adb035ae2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:52.541759 11879 raft_consensus.cc:1261] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b [term 1 FOLLOWER]: Refusing update from remote peer d6032751fcde490e92bedb2adb035ae2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:40:52.542598 11952 consensus_queue.cc:1035] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6caa3d004fb14841a119d40670acbb1b" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 38065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:52.542830 11950 consensus_queue.cc:1035] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b394f4e1e314d2db2a6e736a9c79249" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:40:52.550602 11956 mvcc.cc:205] Tried to move back new op lower bound from 7333601495202701312 to 7333601495023075328. Current Snapshot: MvccSnapshot[applied={T|T < 7333601495202701312}]
I20260926 13:40:52.550820 11957 mvcc.cc:205] Tried to move back new op lower bound from 7333601495202701312 to 7333601495023075328. Current Snapshot: MvccSnapshot[applied={T|T < 7333601495202701312}]
I20260926 13:40:52.560245 11465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:52.560459 11465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:40:52.563979 11465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5467144dba894a309b61c39b67fc984c: Sending DeleteTablet for 3 replicas of tablet 6414ea9130db4a37b363f2d08b661ed8
I20260926 13:40:52.565070 11859 tablet_service.cc:1514] Processing DeleteTablet for tablet 6414ea9130db4a37b363f2d08b661ed8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:52 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:48606
I20260926 13:40:52.565641 11964 tablet_replica.cc:330] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b: stopping tablet replica
I20260926 13:40:52.566062 11964 raft_consensus.cc:2229] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:40:52.566186 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 11528
I20260926 13:40:52.566485 11964 raft_consensus.cc:2258] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:52.567729 11725 tablet_service.cc:1514] Processing DeleteTablet for tablet 6414ea9130db4a37b363f2d08b661ed8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:40:52 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:44902
I20260926 13:40:52.569815 11964 ts_tablet_manager.cc:1905] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:52.572808 11964 ts_tablet_manager.cc:1918] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:52.572924 11964 log.cc:1196] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/6414ea9130db4a37b363f2d08b661ed8
I20260926 13:40:52.573282 11964 ts_tablet_manager.cc:1939] T 6414ea9130db4a37b363f2d08b661ed8 P 6caa3d004fb14841a119d40670acbb1b: Deleting consensus metadata
I20260926 13:40:52.573647 11965 tablet_replica.cc:330] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2: stopping tablet replica
I20260926 13:40:52.574007 11965 raft_consensus.cc:2229] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:40:52.574468 11965 raft_consensus.cc:2258] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:40:52.576262 11452 catalog_manager.cc:4841] TS 6caa3d004fb14841a119d40670acbb1b (127.1.49.3:38065): tablet 6414ea9130db4a37b363f2d08b661ed8 (table test-table [id=009b883803b14ce3a32927d5524e4a83]) successfully deleted
I20260926 13:40:52.576614 11965 ts_tablet_manager.cc:1905] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:40:52.579469 11965 ts_tablet_manager.cc:1918] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:40:52.579595 11965 log.cc:1196] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/6414ea9130db4a37b363f2d08b661ed8
I20260926 13:40:52.579947 11965 ts_tablet_manager.cc:1939] T 6414ea9130db4a37b363f2d08b661ed8 P d6032751fcde490e92bedb2adb035ae2: Deleting consensus metadata
I20260926 13:40:52.581331 11450 catalog_manager.cc:4841] TS d6032751fcde490e92bedb2adb035ae2 (127.1.49.2:34585): tablet 6414ea9130db4a37b363f2d08b661ed8 (table test-table [id=009b883803b14ce3a32927d5524e4a83]) successfully deleted
W20260926 13:40:52.582415 11450 connection.cc:535] client connection to 127.1.49.1:35323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 13:40:52.582595 11450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:40:52.583560 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 11662
I20260926 13:40:52.583833 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:40:52.583500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:35323 (local address 127.0.0.1:37592)
0926 13:40:52.583631 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 13:40:52.583709 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 13:40:52.585435 11450 catalog_manager.cc:4584] TS 9b394f4e1e314d2db2a6e736a9c79249 (127.1.49.1:35323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6414ea9130db4a37b363f2d08b661ed8: Network error: Client connection negotiation failed: client connection to 127.1.49.1:35323: connect: Connection refused (error 111)
I20260926 13:40:52.594300 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 11796
I20260926 13:40:52.602182 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 11434
2026-09-26T13:40:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2519 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:40:52 dist-test-slave-1wns krb5kdc[11969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:40:52 dist-test-slave-1wns krb5kdc[11969](info): set up 2 sockets
Sep 26 13:40:52 dist-test-slave-1wns krb5kdc[11969](info): commencing operation
krb5kdc: starting...
W20260926 13:40:53.697364 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:40:53 dist-test-slave-1wns krb5kdc[11969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:40:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:40:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/rangeradmin_127.1.49.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/rangeradmin_127.1.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/rangerlookup_127.1.49.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/rangerlookup_127.1.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/HTTP_127.1.49.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/HTTP_127.1.49.20@KRBTEST.COM.keytab.
I20260926 13:40:53.886623 1220 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7nlTrX/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/postgres -l logfile start
2026-09-26 13:40:56.863 UTC [11999] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 13:40:56.863 UTC [11999] LOG: listening on IPv4 address "127.1.49.20", port 39273
2026-09-26 13:40:56.868 UTC [11999] LOG: listening on Unix socket "/tmp/.s.PGSQL.39273"
2026-09-26 13:40:56.893 UTC [12002] LOG: database system was shut down at 2026-09-26 13:40:54 UTC
2026-09-26 13:40:56.897 UTC [11999] LOG: database system is ready to accept connections
I20260926 13:40:57.889190 1220 mini_postgres.cc:96] Postgres bound to 39273
2026-09-26 13:40:57.896 UTC [12010] FATAL: database "slave" does not exist
127.1.49.20:39273 - accepting connections
I20260926 13:40:57.896952 1220 mini_ranger.cc:161] Starting Ranger...
I20260926 13:40:57.914757 1220 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 13:40:58.570586 1220 mini_ranger.cc:87] Created ranger Postgres database
I20260926 13:40:58.570716 1220 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-admin
2026-09-26 13:40:58,881 [I] DB FLAVOR :POSTGRES
2026-09-26 13:40:58,882 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:40:58,882 [I] Checking connection..
2026-09-26 13:40:58,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 13:40:59,154 [I] Checking connection passed.
2026-09-26 13:40:59,154 [I] --------- Verifying version history table ---------
2026-09-26 13:40:59,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:40:59,442 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:40:59,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:40:59,721 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:40:59,722 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 13:40:59,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 13:40:59.960 UTC [12074] WARNING: there is no transaction in progress
2026-09-26 13:40:59,967 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 13:40:59,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:41:00,235 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 13:41:00,236 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:41:00,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 13:41:00,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 13:41:00,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1wns.c.gcp-upstream.internal','N') ;"
2026-09-26 13:41:01,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:41:01,320 [I] Table x_portal_user does not exist in database ranger
2026-09-26 13:41:01,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:41:01,602 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 13:41:01,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:41:01,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:41:02,162 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 13:41:02,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 13:41:03.382 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.401 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.421 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.441 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.462 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.484 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.555 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.564 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.584 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.597 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.610 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.629 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.642 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.652 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.661 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:03.672 UTC [12182] WARNING: there is no transaction in progress
2026-09-26 13:41:04,574 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 13:41:04,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 13:41:04,844 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 13:41:04,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:41:05,132 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 13:41:05,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:41:05,427 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 13:41:05,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:41:05,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:41:05,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1wns.c.gcp-upstream.internal';"
2026-09-26 13:41:06,270 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 13:41:06,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 13:41:06,527 [I] DB_PATCHES have already been applied
I20260926 13:41:06.533833 1220 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-admin:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7nlTrX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 13:41:06.533910 1220 mini_ranger.cc:193] Using host: 127.1.49.20
I20260926 13:41:06.537789 1220 mini_ranger.cc:227] Ranger admin URL: http://127.1.49.20:38423
Sep 26, 2026 1:41:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:41:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:41:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 1:41:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 1:41:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 1:41:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 1:41:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 13:41:07.703769 1220 mini_ranger.cc:160] Time spent starting Ranger: real 9.807s user 0.006s sys 0.006s
Sep 26, 2026 1:41:29 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 1:41:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 1:41:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 1:41:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 1:41:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 1:41:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 1:41:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 1:41:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 1:41:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 1:41:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 1:41:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 1:41:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 1:41:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 1:41:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 1:41:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 1:41:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 1:41:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 1:41:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 13:41:31.022637 1220 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/rangerkms_127.1.49.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/rangerkms_127.1.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.49.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.49.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.49.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/HTTP_127.1.49.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.49.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/HTTP_127.1.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 13:41:31 dist-test-slave-1wns krb5kdc[11969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430091, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 13:41:31.141215 1220 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 13:41:31.151474 1220 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 13:41:32.267138 1220 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 13:41:32.267243 1220 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-kms
I20260926 13:41:32.267351 1220 mini_ranger_kms.cc:227] Using postgres at 127.1.49.20:39273
2026-09-26 13:41:32,463 [I] DB FLAVOR :POSTGRES
2026-09-26 13:41:32,463 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:41:32,463 [I] Checking connection
2026-09-26 13:41:32,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 13:41:32,752 [I] connection success
2026-09-26 13:41:32,752 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 13:41:32,752 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 13:41:32,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 13:41:33,080 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 13:41:33,080 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:41:33,080 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 13:41:33,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:39273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 13:41:33,371 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 13:41:33.881001 1220 mini_ranger_kms.cc:310] Created kms service
I20260926 13:41:34.097430 1220 mini_ranger_kms.cc:326] Created kudu user
I20260926 13:41:34.152220 1220 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 13:41:34 dist-test-slave-1wns krb5kdc[11969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430091, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:41:34.616918 1220 mini_ranger_kms.cc:384] Added ranger policy
I20260926 13:41:34.617084 1220 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-kms:/tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 13:41:34.617110 1220 mini_ranger_kms.cc:240] Using host: 127.1.49.20
I20260926 13:41:34.621606 1220 mini_ranger_kms.cc:276] Ranger KMS PID: 12363
I20260926 13:41:34.621776 1220 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.49.20:33937
Sep 26, 2026 1:41:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:41:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:41:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 1:41:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 1:41:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 1:41:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 13:41:36.930860 1220 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.790s user 0.006s sys 0.174s
I20260926 13:41:36.931025 1220 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 13:41:36.931164 1220 mini_ranger_kms.cc:401] 127.1.49.20:33937/kms/v1/keys
Sep 26 13:41:37 dist-test-slave-1wns krb5kdc[11969](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790430097, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.49.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 13:41:37 dist-test-slave-1wns krb5kdc[11969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430097, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.49.20@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
Sep 26 13:41:38 dist-test-slave-1wns krb5kdc[11969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430097, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.49.20@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
Sep 26 13:41:40 dist-test-slave-1wns krb5kdc[11969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:41:40.615537 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:40753
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:40753
--ranger_config_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.49.20:33937/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:41:40.737439 12413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:41:40.737828 12413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:41:40.737910 12413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:41:40.742471 12413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:41:40.742592 12413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:41:40.742631 12413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:41:40.742671 12413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:41:40.742703 12413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:41:40.748852 12413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.49.20:33937/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:40753
--ranger_config_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:40753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:41:40.750137 12413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:41:40.751358 12413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:41:40.762315 12413 server_base.cc:1034] running on GCE node
W20260926 13:41:40.762316 12419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:41:40.762336 12421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:41:40.762336 12418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:41:40.763545 12413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:41:40.765059 12413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:41:40.766320 12413 hybrid_clock.cc:648] HybridClock initialized: now 1790430100766289 us; error 50 us; skew 500 ppm
Sep 26 13:41:40 dist-test-slave-1wns krb5kdc[11969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430100, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:41:40.776657 12413 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:41:40.777802 12413 webserver.cc:460] Webserver started at http://127.1.49.62:40745/ using document root <none> and password file <none>
I20260926 13:41:40.778380 12413 fs_manager.cc:359] Metadata directory not provided
I20260926 13:41:40.778447 12413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:41:40.778647 12413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:41:40 dist-test-slave-1wns krb5kdc[11969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430100, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:41:40.844522 12413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "f5f20266bdd944298944b40ee0a6c885"
format_stamp: "Formatted at 2026-09-26 13:41:40 on dist-test-slave-1wns"
server_key: "8ea7038ca5c88aaeaefd13fa3a7bdfc3"
server_key_iv: "dec58b086cacdf74f8035939679843d0"
server_key_version: "kuduclusterkey@0"
I20260926 13:41:40.845243 12413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "f5f20266bdd944298944b40ee0a6c885"
format_stamp: "Formatted at 2026-09-26 13:41:40 on dist-test-slave-1wns"
server_key: "8ea7038ca5c88aaeaefd13fa3a7bdfc3"
server_key_iv: "dec58b086cacdf74f8035939679843d0"
server_key_version: "kuduclusterkey@0"
I20260926 13:41:40.849701 12413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:41:40.869315 12428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:41:40.871306 12413 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:41:40.871495 12413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "f5f20266bdd944298944b40ee0a6c885"
format_stamp: "Formatted at 2026-09-26 13:41:40 on dist-test-slave-1wns"
server_key: "8ea7038ca5c88aaeaefd13fa3a7bdfc3"
server_key_iv: "dec58b086cacdf74f8035939679843d0"
server_key_version: "kuduclusterkey@0"
I20260926 13:41:40.871667 12413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:41:40.908334 12413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:41:40.912178 12413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:41:40.912405 12413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:41:40.921929 12413 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:40753
I20260926 13:41:40.921959 12480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:40753 every 8 connection(s)
I20260926 13:41:40.923391 12413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:41:40.925431 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 12413
I20260926 13:41:40.925591 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:41:40.925938 1220 external_mini_cluster.cc:1428] Setting key a48d29a68fe2a08484d739d01051f5e9
I20260926 13:41:40.927592 12481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:41:40.935032 12481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885: Bootstrap starting.
Sep 26 13:41:40 dist-test-slave-1wns krb5kdc[11969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:41:40.938434 12481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885: Neither blocks nor log segments found. Creating new log.
I20260926 13:41:40.939666 12481 log.cc:824] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885: Log is configured to *not* fsync() on all Append() calls
I20260926 13:41:40.942229 12481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885: No bootstrap required, opened a new log
I20260926 13:41:40.946003 12484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:40.927657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.1.49.62:40753)
0926 13:41:40.928224 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0926 13:41:40.928247 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 13:41:40.928297 (+ 50us) server_negotiation.cc:407] Connection header received
0926 13:41:40.929255 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:41:40.929284 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:41:40.929856 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:41:40.930379 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:41:40.931366 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:40.933053 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:40.934466 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:40.935229 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:40.935422 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:40.938951 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:41:40.938960 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:41:40.938988 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:41:40.939030 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:41:40.941981 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:41:40.942652 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:41:40.942658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:41:40.942666 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:41:40.942816 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:41:40.943183 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:41:40.943186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:41:40.943188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:41:40.943774 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:41:40.944028 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 13:41:40.944388 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 13:41:40.944903 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":170,"threads_started":1}
I20260926 13:41:40.946359 12481 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f20266bdd944298944b40ee0a6c885" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40753 } }
I20260926 13:41:40.946688 12481 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:41:40.946834 12481 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f20266bdd944298944b40ee0a6c885, State: Initialized, Role: FOLLOWER
I20260926 13:41:40.947458 12481 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f20266bdd944298944b40ee0a6c885" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40753 } }
I20260926 13:41:40.947636 12481 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:41:40.947737 12481 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:41:40.947896 12481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:41:40.949245 12481 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f20266bdd944298944b40ee0a6c885" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40753 } }
I20260926 13:41:40.949651 12481 leader_election.cc:302] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5f20266bdd944298944b40ee0a6c885; no voters:
I20260926 13:41:40.950037 12481 leader_election.cc:288] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:41:40.950182 12486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:41:40.950430 12486 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [term 1 LEADER]: Becoming Leader. State: Replica: f5f20266bdd944298944b40ee0a6c885, State: Running, Role: LEADER
I20260926 13:41:40.950850 12486 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f20266bdd944298944b40ee0a6c885" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40753 } }
I20260926 13:41:40.951498 12481 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:41:40.952762 12488 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5f20266bdd944298944b40ee0a6c885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f20266bdd944298944b40ee0a6c885" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40753 } } }
I20260926 13:41:40.952905 12488 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [sys.catalog]: This master's current role is: LEADER
I20260926 13:41:40.953810 12487 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5f20266bdd944298944b40ee0a6c885. Latest consensus state: current_term: 1 leader_uuid: "f5f20266bdd944298944b40ee0a6c885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f20266bdd944298944b40ee0a6c885" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40753 } } }
I20260926 13:41:40.953979 12487 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885 [sys.catalog]: This master's current role is: LEADER
I20260926 13:41:40.954276 12481 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 13:41:40.955717 12495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:41:40.964459 12495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:41:40.976424 12495 catalog_manager.cc:1347] Generated new cluster ID: 36150221d29f46bdb61d2c3d228e08ae
I20260926 13:41:40.976507 12495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:41:41.009385 12495 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:41:41.012005 12495 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:41:41.018616 12495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885: Generated new TSK 0
I20260926 13:41:41.019860 12495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:41:41.789249 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.49.20:33937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40753
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:41:41.956208 12526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:41:41.956670 12526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:41:41.956744 12526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:41:41.964007 12526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:41:41.964093 12526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:41:41.964198 12526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:41:41.972074 12526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.49.20:33937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:41:41.973815 12526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:41:41.975320 12526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:41:41.989960 12526 server_base.cc:1034] running on GCE node
W20260926 13:41:41.989913 12533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:41:41.990140 12534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:41:41.990172 12536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:41:41.990736 12526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:41:41.991477 12526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:41:41.992679 12526 hybrid_clock.cc:648] HybridClock initialized: now 1790430101992643 us; error 61 us; skew 500 ppm
Sep 26 13:41:41 dist-test-slave-1wns krb5kdc[11969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430101, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:41:41.995893 12526 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:41:41.997095 12526 webserver.cc:460] Webserver started at http://127.1.49.1:36569/ using document root <none> and password file <none>
I20260926 13:41:41.998057 12526 fs_manager.cc:359] Metadata directory not provided
I20260926 13:41:41.998193 12526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:41:41.998484 12526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:41:41 dist-test-slave-1wns krb5kdc[11969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430101, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:41:42.029816 12526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "59e6077966b147258a3d768b0820a96d"
format_stamp: "Formatted at 2026-09-26 13:41:42 on dist-test-slave-1wns"
server_key: "ae5f071076308be5c15a74480b687f92"
server_key_iv: "87751a68599e56b652329aeb73eab698"
server_key_version: "kuduclusterkey@0"
I20260926 13:41:42.030681 12526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "59e6077966b147258a3d768b0820a96d"
format_stamp: "Formatted at 2026-09-26 13:41:42 on dist-test-slave-1wns"
server_key: "ae5f071076308be5c15a74480b687f92"
server_key_iv: "87751a68599e56b652329aeb73eab698"
server_key_version: "kuduclusterkey@0"
I20260926 13:41:42.037870 12526 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 13:41:42.075325 12545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:41:42.077315 12526 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:41:42.077488 12526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "59e6077966b147258a3d768b0820a96d"
format_stamp: "Formatted at 2026-09-26 13:41:42 on dist-test-slave-1wns"
server_key: "ae5f071076308be5c15a74480b687f92"
server_key_iv: "87751a68599e56b652329aeb73eab698"
server_key_version: "kuduclusterkey@0"
I20260926 13:41:42.077654 12526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:41:42.107456 12526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:41:42.111929 12526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:41:42.112160 12526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:41:42.112946 12526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:41:42.114164 12526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:41:42.114241 12526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:41:42.114310 12526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:41:42.114387 12526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 13:41:42.127944 12526 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:44407
I20260926 13:41:42.128286 12658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:44407 every 8 connection(s)
I20260926 13:41:42.129347 12526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:41:42.130111 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 12526
I20260926 13:41:42.130250 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:41:42.130673 1220 external_mini_cluster.cc:1428] Setting key 84752d3a5c1aa1cfeb705e62214255b8
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
Sep 26 13:41:42 dist-test-slave-1wns krb5kdc[11969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430101, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
I20260926 13:41:42.156069 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:42.132809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:54227 (local address 127.1.49.62:40753)
0926 13:41:42.135589 (+ 2780us) server_negotiation.cc:206] Beginning negotiation
0926 13:41:42.135595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:41:42.136201 (+ 606us) server_negotiation.cc:407] Connection header received
0926 13:41:42.137724 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:41:42.137728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:41:42.137798 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:41:42.137918 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:41:42.139903 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.140803 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.143039 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.143454 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.143596 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:42.150446 (+ 6850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:41:42.150452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:41:42.150456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:41:42.150493 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:41:42.152877 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:41:42.153778 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:41:42.153782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:41:42.153784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:41:42.153894 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:41:42.154436 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:41:42.154440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:41:42.154442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:41:42.154731 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:41:42.154832 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 13:41:42.155672 (+ 840us) server_negotiation.cc:299] Negotiation successful
0926 13:41:42.155859 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2673,"thread_start_us":82,"threads_started":1}
I20260926 13:41:42.157975 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:42.133094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:40753 (local address 127.1.49.1:54227)
0926 13:41:42.135976 (+ 2882us) negotiation.cc:107] Waiting for socket to connect
0926 13:41:42.136062 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0926 13:41:42.137393 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:41:42.138315 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:41:42.138328 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:41:42.138789 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:41:42.139613 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:42.139639 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.141024 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.141028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:42.142838 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:42.142853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.143897 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.143902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:42.144155 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:42.144993 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:41:42.145020 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:41:42.150127 (+ 5107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:41:42.153120 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:41:42.153130 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:41:42.153153 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:41:42.153626 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:41:42.154025 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:41:42.154028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:41:42.154030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:41:42.154301 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:41:42.154913 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:41:42.154922 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:41:42.155326 (+ 404us) client_negotiation.cc:769] Sending connection context
0926 13:41:42.156361 (+ 1035us) client_negotiation.cc:240] Negotiation successful
0926 13:41:42.156770 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2570,"thread_start_us":168,"threads_started":1}
I20260926 13:41:42.159600 12659 heartbeater.cc:347] Connected to a master server at 127.1.49.62:40753
I20260926 13:41:42.159927 12659 heartbeater.cc:464] Registering TS with master...
I20260926 13:41:42.160617 12659 heartbeater.cc:511] Master 127.1.49.62:40753 requested a full tablet report, sending...
I20260926 13:41:42.162662 12438 ts_manager.cc:194] Registered new tserver with Master: 59e6077966b147258a3d768b0820a96d (127.1.49.1:44407)
I20260926 13:41:42.164438 12438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:54227
Sep 26 13:41:42 dist-test-slave-1wns krb5kdc[11969](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790430102, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:41:42.179570 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.49.20:33937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40753
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:41:42.334829 12665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:41:42.335199 12665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:41:42.335356 12665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:41:42.342296 12665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:41:42.342494 12665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:41:42.342725 12665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:41:42.349772 12665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.49.20:33937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:41:42.351099 12665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:41:42.352651 12665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:41:42.363992 12672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:41:42.363936 12674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:41:42.364207 12671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:41:42.364342 12665 server_base.cc:1034] running on GCE node
I20260926 13:41:42.365130 12665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:41:42.365981 12665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:41:42.367199 12665 hybrid_clock.cc:648] HybridClock initialized: now 1790430102367145 us; error 78 us; skew 500 ppm
Sep 26 13:41:42 dist-test-slave-1wns krb5kdc[11969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430102, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:41:42.370524 12665 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:41:42.371672 12665 webserver.cc:460] Webserver started at http://127.1.49.2:32865/ using document root <none> and password file <none>
I20260926 13:41:42.372363 12665 fs_manager.cc:359] Metadata directory not provided
I20260926 13:41:42.372490 12665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:41:42.372745 12665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:41:42 dist-test-slave-1wns krb5kdc[11969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430102, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:41:42.394932 12665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "10d6c06c4d55443fb1848addde66c9e8"
format_stamp: "Formatted at 2026-09-26 13:41:42 on dist-test-slave-1wns"
server_key: "90dbbecedf92d164b0213d381f4fb7d8"
server_key_iv: "0917d87a4c1d12556e607bbc8a312ac8"
server_key_version: "kuduclusterkey@0"
I20260926 13:41:42.395907 12665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "10d6c06c4d55443fb1848addde66c9e8"
format_stamp: "Formatted at 2026-09-26 13:41:42 on dist-test-slave-1wns"
server_key: "90dbbecedf92d164b0213d381f4fb7d8"
server_key_iv: "0917d87a4c1d12556e607bbc8a312ac8"
server_key_version: "kuduclusterkey@0"
I20260926 13:41:42.400897 12665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 13:41:42.428710 12681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:41:42.430657 12665 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260926 13:41:42.430927 12665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "10d6c06c4d55443fb1848addde66c9e8"
format_stamp: "Formatted at 2026-09-26 13:41:42 on dist-test-slave-1wns"
server_key: "90dbbecedf92d164b0213d381f4fb7d8"
server_key_iv: "0917d87a4c1d12556e607bbc8a312ac8"
server_key_version: "kuduclusterkey@0"
I20260926 13:41:42.431185 12665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:41:42.448798 12665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:41:42.452461 12665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:41:42.452699 12665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:41:42.453536 12665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:41:42.454591 12665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:41:42.454675 12665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:41:42.454746 12665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:41:42.454777 12665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:41:42.466915 12665 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:46451
I20260926 13:41:42.466991 12794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:46451 every 8 connection(s)
I20260926 13:41:42.468107 12665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:41:42.473244 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 12665
I20260926 13:41:42.473424 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:41:42.473839 1220 external_mini_cluster.cc:1428] Setting key baf194e4f5b8fb4e9a0b171235659df2
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
Sep 26 13:41:42 dist-test-slave-1wns krb5kdc[11969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430102, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:41:42.510874 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:42.470196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:55355 (local address 127.1.49.62:40753)
0926 13:41:42.470366 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 13:41:42.470371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:41:42.471518 (+ 1147us) server_negotiation.cc:407] Connection header received
0926 13:41:42.473611 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:41:42.473616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:41:42.473689 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:41:42.473800 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:41:42.488481 (+ 14681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.489416 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.494265 (+ 4849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.494716 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.494842 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:42.500592 (+ 5750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:41:42.500600 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:41:42.500607 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:41:42.500650 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:41:42.502666 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:41:42.506755 (+ 4089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:41:42.506762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:41:42.506766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:41:42.506892 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:41:42.507470 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:41:42.507473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:41:42.507475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:41:42.507758 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:41:42.507894 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 13:41:42.510502 (+ 2608us) server_negotiation.cc:299] Negotiation successful
0926 13:41:42.510684 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 13:41:42.512969 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:42.470686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:40753 (local address 127.1.49.2:55355)
0926 13:41:42.471291 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 13:41:42.471372 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 13:41:42.473224 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:41:42.474098 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:41:42.474112 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:41:42.475026 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:41:42.488180 (+ 13154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:42.488231 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.489655 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.489660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:42.491321 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:42.491337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.495144 (+ 3807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.495149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:42.495574 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:42.496458 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:41:42.496490 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:41:42.500312 (+ 3822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:41:42.505766 (+ 5454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:41:42.505776 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:41:42.505800 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:41:42.506379 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:41:42.507046 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:41:42.507051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:41:42.507054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:41:42.507329 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:41:42.507969 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:41:42.507977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:41:42.508423 (+ 446us) client_negotiation.cc:769] Sending connection context
0926 13:41:42.511159 (+ 2736us) client_negotiation.cc:240] Negotiation successful
0926 13:41:42.511615 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":139,"threads_started":1}
I20260926 13:41:42.514782 12795 heartbeater.cc:347] Connected to a master server at 127.1.49.62:40753
I20260926 13:41:42.515062 12795 heartbeater.cc:464] Registering TS with master...
I20260926 13:41:42.515735 12795 heartbeater.cc:511] Master 127.1.49.62:40753 requested a full tablet report, sending...
I20260926 13:41:42.517226 12438 ts_manager.cc:194] Registered new tserver with Master: 10d6c06c4d55443fb1848addde66c9e8 (127.1.49.2:46451)
I20260926 13:41:42.518167 12438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:55355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:41:42.524184 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.49.20:33937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40753
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:41:42 dist-test-slave-1wns krb5kdc[11969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430102, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
W20260926 13:41:42.681826 12800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:41:42.682380 12800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:41:42.682504 12800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:41:42.689978 12800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:41:42.690188 12800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:41:42.690346 12800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:41:42.702231 12800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.49.20:33937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:40753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:41:42.703862 12800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:41:42.705464 12800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:41:42.720346 12808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:41:42.720680 12805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:41:42.721093 12806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:41:42.721907 12800 server_base.cc:1034] running on GCE node
I20260926 13:41:42.722525 12800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:41:42.723417 12800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:41:42.730978 12800 hybrid_clock.cc:648] HybridClock initialized: now 1790430102730954 us; error 129 us; skew 500 ppm
Sep 26 13:41:42 dist-test-slave-1wns krb5kdc[11969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430102, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:41:42.744159 12800 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:41:42.745469 12800 webserver.cc:460] Webserver started at http://127.1.49.3:35051/ using document root <none> and password file <none>
I20260926 13:41:42.746160 12800 fs_manager.cc:359] Metadata directory not provided
I20260926 13:41:42.746269 12800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:41:42.746595 12800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:41:42 dist-test-slave-1wns krb5kdc[11969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430102, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:41:42.773910 12800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "527764a19aea4b0098306a19b44bcc89"
format_stamp: "Formatted at 2026-09-26 13:41:42 on dist-test-slave-1wns"
server_key: "e64c591b661afdacdccbd0e35a338df7"
server_key_iv: "0801b6995e1fd385b1f862b4ff1bd272"
server_key_version: "kuduclusterkey@0"
I20260926 13:41:42.774689 12800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "527764a19aea4b0098306a19b44bcc89"
format_stamp: "Formatted at 2026-09-26 13:41:42 on dist-test-slave-1wns"
server_key: "e64c591b661afdacdccbd0e35a338df7"
server_key_iv: "0801b6995e1fd385b1f862b4ff1bd272"
server_key_version: "kuduclusterkey@0"
I20260926 13:41:42.780375 12800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 13:41:42.798007 12815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:41:42.800050 12800 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:41:42.800243 12800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "527764a19aea4b0098306a19b44bcc89"
format_stamp: "Formatted at 2026-09-26 13:41:42 on dist-test-slave-1wns"
server_key: "e64c591b661afdacdccbd0e35a338df7"
server_key_iv: "0801b6995e1fd385b1f862b4ff1bd272"
server_key_version: "kuduclusterkey@0"
I20260926 13:41:42.800397 12800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:41:42.817296 12800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:41:42.821458 12800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:41:42.821668 12800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:41:42.822393 12800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:41:42.823578 12800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:41:42.823643 12800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:41:42.823700 12800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:41:42.823776 12800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 13:41:42.835850 12800 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:44275
I20260926 13:41:42.837294 12800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:41:42.843638 12928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:44275 every 8 connection(s)
I20260926 13:41:42.845969 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 12800
I20260926 13:41:42.846119 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:41:42.846494 1220 external_mini_cluster.cc:1428] Setting key cc6673314c30d786f6e1fac97019a7dd
Sep 26 13:41:42 dist-test-slave-1wns krb5kdc[11969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430102, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:41:42.856685 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:42.839743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:59535 (local address 127.1.49.62:40753)
0926 13:41:42.839913 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 13:41:42.839918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:41:42.841341 (+ 1423us) server_negotiation.cc:407] Connection header received
0926 13:41:42.842749 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:41:42.842753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:41:42.842822 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:41:42.842927 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:41:42.844723 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.845582 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.847784 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.848164 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.848249 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:42.851699 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:41:42.851704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:41:42.851707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:41:42.851738 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:41:42.853737 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:41:42.854536 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:41:42.854540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:41:42.854542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:41:42.854654 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:41:42.855155 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:41:42.855159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:41:42.855161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:41:42.855405 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:41:42.855510 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 13:41:42.856353 (+ 843us) server_negotiation.cc:299] Negotiation successful
0926 13:41:42.856502 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 13:41:42.858358 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:42.840418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:40753 (local address 127.1.49.3:59535)
0926 13:41:42.841128 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 13:41:42.841202 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 13:41:42.842398 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:41:42.843297 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:41:42.843307 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:41:42.843714 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:41:42.844475 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:42.844498 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.845805 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.845808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:42.847624 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:42.847638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.848478 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.848482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:42.848719 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:42.849507 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:41:42.849529 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:41:42.851448 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:41:42.853897 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:41:42.853907 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:41:42.853929 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:41:42.854388 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:41:42.854779 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:41:42.854785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:41:42.854787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:41:42.855025 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:41:42.855544 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:41:42.855552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:41:42.856007 (+ 455us) client_negotiation.cc:769] Sending connection context
0926 13:41:42.856281 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 13:41:42.856682 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":209,"threads_started":1}
I20260926 13:41:42.859863 12929 heartbeater.cc:347] Connected to a master server at 127.1.49.62:40753
I20260926 13:41:42.860146 12929 heartbeater.cc:464] Registering TS with master...
I20260926 13:41:42.860761 12929 heartbeater.cc:511] Master 127.1.49.62:40753 requested a full tablet report, sending...
I20260926 13:41:42.862003 12438 ts_manager.cc:194] Registered new tserver with Master: 527764a19aea4b0098306a19b44bcc89 (127.1.49.3:44275)
I20260926 13:41:42.862818 12438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:59535
I20260926 13:41:42.873880 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:41:42.888566 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:42.876991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49720 (local address 127.1.49.62:40753)
0926 13:41:42.877161 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 13:41:42.877168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:41:42.877769 (+ 601us) server_negotiation.cc:407] Connection header received
0926 13:41:42.878006 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:41:42.878010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:41:42.878067 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:41:42.878158 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:41:42.879569 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.880540 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.881849 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.882207 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.882300 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:42.884373 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:41:42.884377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:41:42.884380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:41:42.884417 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:41:42.886389 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:41:42.887046 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:41:42.887050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:41:42.887053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:41:42.887164 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:41:42.887489 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:41:42.887493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:41:42.887495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:41:42.887726 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:41:42.887857 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 13:41:42.888258 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 13:41:42.888397 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 13:41:42.893342 12441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:41:42.897766 12441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:41:42.920151 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:42.911694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:44275 (local address 127.0.0.1:42108)
0926 13:41:42.912295 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 13:41:42.912358 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 13:41:42.912500 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:41:42.915102 (+ 2602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:41:42.915109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:41:42.915146 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:41:42.915306 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:42.915312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.916810 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.916814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:42.918270 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:42.918282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.919486 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.919490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:42.919637 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:42.919687 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 13:41:42.919788 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 13:41:42.919955 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":207,"threads_started":1}
I20260926 13:41:42.921715 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:42.912792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.1.49.3:44275)
0926 13:41:42.914483 (+ 1691us) server_negotiation.cc:206] Beginning negotiation
0926 13:41:42.914520 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0926 13:41:42.914549 (+ 29us) server_negotiation.cc:407] Connection header received
0926 13:41:42.914640 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:41:42.914647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:41:42.914901 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:41:42.915083 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:41:42.915447 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.916667 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.918466 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.919335 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.920858 (+ 1523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:42.921345 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0926 13:41:42.921399 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 13:41:42.921514 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":170,"threads_started":1}
I20260926 13:41:42.925323 12852 tablet_service.cc:1467] Processing CreateTablet for tablet d5512f992810448b887bfb8d60487716 (DEFAULT_TABLE table=test-table [id=19ca67921c3f4d7dbef54c44ab0c70f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:41:42.926998 12852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5512f992810448b887bfb8d60487716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:41:42.938030 12947 tablet_bootstrap.cc:492] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89: Bootstrap starting.
I20260926 13:41:42.941277 12947 tablet_bootstrap.cc:654] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89: Neither blocks nor log segments found. Creating new log.
I20260926 13:41:42.942142 12947 log.cc:824] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89: Log is configured to *not* fsync() on all Append() calls
I20260926 13:41:42.944977 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:42.913926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:46451 (local address 127.0.0.1:52672)
0926 13:41:42.919982 (+ 6056us) negotiation.cc:107] Waiting for socket to connect
0926 13:41:42.919998 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:41:42.920103 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:41:42.931906 (+ 11803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:41:42.931910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:41:42.931944 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:41:42.932070 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:42.932077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.935783 (+ 3706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.935787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:42.937045 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:42.937059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.944507 (+ 7448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.944511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:42.944622 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:42.944656 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 13:41:42.944723 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:41:42.944814 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5982,"thread_start_us":98,"threads_started":1}
I20260926 13:41:42.945287 12947 tablet_bootstrap.cc:492] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89: No bootstrap required, opened a new log
I20260926 13:41:42.945572 12947 ts_tablet_manager.cc:1397] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 13:41:42.946153 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:42.914412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.1.49.2:46451)
0926 13:41:42.921162 (+ 6750us) server_negotiation.cc:206] Beginning negotiation
0926 13:41:42.921175 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:41:42.921202 (+ 27us) server_negotiation.cc:407] Connection header received
0926 13:41:42.921281 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:41:42.921287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:41:42.921497 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:41:42.921666 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:41:42.934504 (+ 12838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.935606 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.943282 (+ 7676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.944323 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.945580 (+ 1257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:42.945813 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 13:41:42.945880 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 13:41:42.945989 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6622,"thread_start_us":136,"threads_started":1}
I20260926 13:41:42.946925 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:42.913754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:44407 (local address 127.0.0.1:52630)
0926 13:41:42.914479 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 13:41:42.914495 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:41:42.914602 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:41:42.932276 (+ 17674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:41:42.932279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:41:42.932298 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:41:42.932391 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:42.932397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.944083 (+ 11686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.944087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:42.945246 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:42.945260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.946508 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.946512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:42.946617 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:42.946645 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 13:41:42.946700 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 13:41:42.946778 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":87,"threads_started":1}
I20260926 13:41:42.947618 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:42.913920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.1.49.1:44407)
0926 13:41:42.920387 (+ 6467us) server_negotiation.cc:206] Beginning negotiation
0926 13:41:42.920394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:41:42.920422 (+ 28us) server_negotiation.cc:407] Connection header received
0926 13:41:42.920513 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:41:42.920519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:41:42.920806 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:41:42.932569 (+ 11763us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:41:42.932746 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.934279 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.945453 (+ 11174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:42.946342 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:42.947140 (+ 798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:42.947312 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 13:41:42.947372 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 13:41:42.947474 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6340,"thread_start_us":114,"threads_started":1}
I20260926 13:41:42.949941 12947 raft_consensus.cc:348] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527764a19aea4b0098306a19b44bcc89" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44275 } } peers { permanent_uuid: "59e6077966b147258a3d768b0820a96d" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44407 } } peers { permanent_uuid: "10d6c06c4d55443fb1848addde66c9e8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46451 } }
I20260926 13:41:42.950533 12947 raft_consensus.cc:374] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:41:42.950659 12947 raft_consensus.cc:729] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527764a19aea4b0098306a19b44bcc89, State: Initialized, Role: FOLLOWER
I20260926 13:41:42.951279 12947 consensus_queue.cc:260] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527764a19aea4b0098306a19b44bcc89" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44275 } } peers { permanent_uuid: "59e6077966b147258a3d768b0820a96d" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44407 } } peers { permanent_uuid: "10d6c06c4d55443fb1848addde66c9e8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46451 } }
I20260926 13:41:42.952639 12947 ts_tablet_manager.cc:1428] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20260926 13:41:42.953112 12929 heartbeater.cc:503] Master 127.1.49.62:40753 was elected leader, sending a full tablet report...
I20260926 13:41:42.955353 12575 tablet_service.cc:1467] Processing CreateTablet for tablet d5512f992810448b887bfb8d60487716 (DEFAULT_TABLE table=test-table [id=19ca67921c3f4d7dbef54c44ab0c70f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:41:42.950574 12729 tablet_service.cc:1467] Processing CreateTablet for tablet d5512f992810448b887bfb8d60487716 (DEFAULT_TABLE table=test-table [id=19ca67921c3f4d7dbef54c44ab0c70f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:41:42.957058 12575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5512f992810448b887bfb8d60487716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:41:42.957903 12729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5512f992810448b887bfb8d60487716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:41:42.972344 12950 tablet_bootstrap.cc:492] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d: Bootstrap starting.
I20260926 13:41:42.975019 12950 tablet_bootstrap.cc:654] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d: Neither blocks nor log segments found. Creating new log.
I20260926 13:41:42.976009 12950 log.cc:824] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:41:42.977435 12951 tablet_bootstrap.cc:492] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8: Bootstrap starting.
I20260926 13:41:42.978981 12950 tablet_bootstrap.cc:492] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d: No bootstrap required, opened a new log
I20260926 13:41:42.979244 12950 ts_tablet_manager.cc:1397] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260926 13:41:42.980305 12951 tablet_bootstrap.cc:654] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8: Neither blocks nor log segments found. Creating new log.
I20260926 13:41:42.981782 12951 log.cc:824] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:41:42.984441 12951 tablet_bootstrap.cc:492] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8: No bootstrap required, opened a new log
I20260926 13:41:42.984222 12950 raft_consensus.cc:348] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527764a19aea4b0098306a19b44bcc89" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44275 } } peers { permanent_uuid: "59e6077966b147258a3d768b0820a96d" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44407 } } peers { permanent_uuid: "10d6c06c4d55443fb1848addde66c9e8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46451 } }
I20260926 13:41:42.984527 12950 raft_consensus.cc:374] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:41:42.984577 12950 raft_consensus.cc:729] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e6077966b147258a3d768b0820a96d, State: Initialized, Role: FOLLOWER
I20260926 13:41:42.984674 12951 ts_tablet_manager.cc:1397] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 13:41:42.985121 12950 consensus_queue.cc:260] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527764a19aea4b0098306a19b44bcc89" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44275 } } peers { permanent_uuid: "59e6077966b147258a3d768b0820a96d" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44407 } } peers { permanent_uuid: "10d6c06c4d55443fb1848addde66c9e8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46451 } }
I20260926 13:41:42.986263 12950 ts_tablet_manager.cc:1428] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 13:41:42.986696 12659 heartbeater.cc:503] Master 127.1.49.62:40753 was elected leader, sending a full tablet report...
I20260926 13:41:42.989319 12951 raft_consensus.cc:348] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527764a19aea4b0098306a19b44bcc89" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44275 } } peers { permanent_uuid: "59e6077966b147258a3d768b0820a96d" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44407 } } peers { permanent_uuid: "10d6c06c4d55443fb1848addde66c9e8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46451 } }
I20260926 13:41:42.989797 12951 raft_consensus.cc:374] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:41:42.989872 12951 raft_consensus.cc:729] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d6c06c4d55443fb1848addde66c9e8, State: Initialized, Role: FOLLOWER
I20260926 13:41:42.990496 12951 consensus_queue.cc:260] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527764a19aea4b0098306a19b44bcc89" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44275 } } peers { permanent_uuid: "59e6077966b147258a3d768b0820a96d" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44407 } } peers { permanent_uuid: "10d6c06c4d55443fb1848addde66c9e8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46451 } }
I20260926 13:41:42.991611 12795 heartbeater.cc:503] Master 127.1.49.62:40753 was elected leader, sending a full tablet report...
I20260926 13:41:42.991838 12951 ts_tablet_manager.cc:1428] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
Sep 26 13:41:43 dist-test-slave-1wns krb5kdc[11969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430102, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:41:43.067788 12949 raft_consensus.cc:482] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:41:43.068060 12949 raft_consensus.cc:504] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527764a19aea4b0098306a19b44bcc89" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44275 } } peers { permanent_uuid: "59e6077966b147258a3d768b0820a96d" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44407 } } peers { permanent_uuid: "10d6c06c4d55443fb1848addde66c9e8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46451 } }
I20260926 13:41:43.069289 12949 leader_election.cc:288] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59e6077966b147258a3d768b0820a96d (127.1.49.1:44407), 10d6c06c4d55443fb1848addde66c9e8 (127.1.49.2:46451)
I20260926 13:41:43.074833 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:43.069484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:44407 (local address 127.1.49.3:46627)
0926 13:41:43.069929 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 13:41:43.069947 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:41:43.070047 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:41:43.070686 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:41:43.070692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:41:43.070717 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:41:43.070856 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:43.070863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:43.071962 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:43.071966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:43.073279 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:43.073292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:43.074366 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:43.074370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:43.074478 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:43.074502 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 13:41:43.074583 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 13:41:43.074667 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342}
I20260926 13:41:43.075825 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:43.070216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:46627 (local address 127.1.49.1:44407)
0926 13:41:43.070358 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 13:41:43.070362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:41:43.070377 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:41:43.070433 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:41:43.070435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:41:43.070486 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:41:43.070561 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:41:43.070991 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:43.071752 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:43.073442 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:43.074098 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:43.074224 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:43.074303 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:41:43.075554 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0926 13:41:43.075642 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:41:43.076720 12613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5512f992810448b887bfb8d60487716" candidate_uuid: "527764a19aea4b0098306a19b44bcc89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e6077966b147258a3d768b0820a96d" is_pre_election: true
I20260926 13:41:43.077088 12613 raft_consensus.cc:2454] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 527764a19aea4b0098306a19b44bcc89 in term 0.
I20260926 13:41:43.077720 12817 leader_election.cc:302] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 527764a19aea4b0098306a19b44bcc89, 59e6077966b147258a3d768b0820a96d; no voters:
I20260926 13:41:43.078130 12949 raft_consensus.cc:2781] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:41:43.078254 12949 raft_consensus.cc:482] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:41:43.078305 12949 raft_consensus.cc:3037] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:41:43.079362 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:43.070931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:46451 (local address 127.1.49.3:42357)
0926 13:41:43.075064 (+ 4133us) negotiation.cc:107] Waiting for socket to connect
0926 13:41:43.075080 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:41:43.075177 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:41:43.075537 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:41:43.075539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:41:43.075558 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:41:43.075635 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:43.075640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:43.076713 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:43.076716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:43.077826 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:41:43.077837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:41:43.078913 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:41:43.078917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:41:43.079011 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:43.079034 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:41:43.079119 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:41:43.079199 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4062,"thread_start_us":132,"threads_started":1}
I20260926 13:41:43.079731 12949 raft_consensus.cc:504] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527764a19aea4b0098306a19b44bcc89" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44275 } } peers { permanent_uuid: "59e6077966b147258a3d768b0820a96d" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44407 } } peers { permanent_uuid: "10d6c06c4d55443fb1848addde66c9e8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46451 } }
I20260926 13:41:43.080139 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:43.071362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:42357 (local address 127.1.49.2:46451)
0926 13:41:43.071476 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 13:41:43.071480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:41:43.075138 (+ 3658us) server_negotiation.cc:407] Connection header received
0926 13:41:43.075265 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:41:43.075268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:41:43.075351 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:41:43.075418 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:41:43.075793 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:43.076569 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:43.078004 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:43.078660 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:43.078778 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:43.078848 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:41:43.079929 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0926 13:41:43.080024 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:41:43.080235 12949 leader_election.cc:288] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [CANDIDATE]: Term 1 election: Requested vote from peers 59e6077966b147258a3d768b0820a96d (127.1.49.1:44407), 10d6c06c4d55443fb1848addde66c9e8 (127.1.49.2:46451)
I20260926 13:41:43.080812 12749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5512f992810448b887bfb8d60487716" candidate_uuid: "527764a19aea4b0098306a19b44bcc89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d6c06c4d55443fb1848addde66c9e8" is_pre_election: true
I20260926 13:41:43.081151 12749 raft_consensus.cc:2454] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 527764a19aea4b0098306a19b44bcc89 in term 0.
I20260926 13:41:43.081343 12613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5512f992810448b887bfb8d60487716" candidate_uuid: "527764a19aea4b0098306a19b44bcc89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e6077966b147258a3d768b0820a96d"
I20260926 13:41:43.081491 12613 raft_consensus.cc:3037] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:41:43.081552 12748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5512f992810448b887bfb8d60487716" candidate_uuid: "527764a19aea4b0098306a19b44bcc89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d6c06c4d55443fb1848addde66c9e8"
I20260926 13:41:43.081686 12748 raft_consensus.cc:3037] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:41:43.082836 12748 raft_consensus.cc:2454] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527764a19aea4b0098306a19b44bcc89 in term 1.
I20260926 13:41:43.083210 12819 leader_election.cc:302] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10d6c06c4d55443fb1848addde66c9e8, 527764a19aea4b0098306a19b44bcc89; no voters:
I20260926 13:41:43.083381 12613 raft_consensus.cc:2454] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527764a19aea4b0098306a19b44bcc89 in term 1.
I20260926 13:41:43.084259 12949 raft_consensus.cc:2781] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:41:43.084520 12949 raft_consensus.cc:686] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [term 1 LEADER]: Becoming Leader. State: Replica: 527764a19aea4b0098306a19b44bcc89, State: Running, Role: LEADER
I20260926 13:41:43.084951 12949 consensus_queue.cc:237] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527764a19aea4b0098306a19b44bcc89" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44275 } } peers { permanent_uuid: "59e6077966b147258a3d768b0820a96d" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44407 } } peers { permanent_uuid: "10d6c06c4d55443fb1848addde66c9e8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46451 } }
I20260926 13:41:43.089038 12441 catalog_manager.cc:5495] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 527764a19aea4b0098306a19b44bcc89 (127.1.49.3). New cstate: current_term: 1 leader_uuid: "527764a19aea4b0098306a19b44bcc89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527764a19aea4b0098306a19b44bcc89" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59e6077966b147258a3d768b0820a96d" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10d6c06c4d55443fb1848addde66c9e8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46451 } health_report { overall_health: UNKNOWN } } }
W20260926 13:41:43.093485 12930 tablet.cc:2369] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:41:43.116669 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:43.110963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.1.49.3:44275)
0926 13:41:43.111088 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 13:41:43.111093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:41:43.111108 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:41:43.111208 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:41:43.111211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:41:43.111302 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:41:43.111425 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:41:43.111949 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:43.112726 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:43.114084 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:41:43.114449 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:41:43.114568 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:41:43.115009 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:41:43.115682 (+ 673us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:41:43.115851 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 13:41:43.116412 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 13:41:43.116491 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 13:41:43.133234 12660 tablet.cc:2369] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:41:43.144833 12613 raft_consensus.cc:1261] T d5512f992810448b887bfb8d60487716 P 59e6077966b147258a3d768b0820a96d [term 1 FOLLOWER]: Refusing update from remote peer 527764a19aea4b0098306a19b44bcc89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:41:43.144837 12748 raft_consensus.cc:1261] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8 [term 1 FOLLOWER]: Refusing update from remote peer 527764a19aea4b0098306a19b44bcc89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:41:43.145824 12959 consensus_queue.cc:1035] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10d6c06c4d55443fb1848addde66c9e8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:41:43.146447 12959 consensus_queue.cc:1035] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59e6077966b147258a3d768b0820a96d" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:41:43.162287 12962 mvcc.cc:205] Tried to move back new op lower bound from 7333601702471462912 to 7333601702239879168. Current Snapshot: MvccSnapshot[applied={T|T < 7333601702471462912}]
I20260926 13:41:43.164153 12965 mvcc.cc:205] Tried to move back new op lower bound from 7333601702471462912 to 7333601702239879168. Current Snapshot: MvccSnapshot[applied={T|T < 7333601702471462912}]
I20260926 13:41:43.180164 12963 mvcc.cc:205] Tried to move back new op lower bound from 7333601702471462912 to 7333601702239879168. Current Snapshot: MvccSnapshot[applied={T|T < 7333601702471462912}]
I20260926 13:41:43.182482 12441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:41:43.183105 12441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:41:43.186748 12441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5f20266bdd944298944b40ee0a6c885: Sending DeleteTablet for 3 replicas of tablet d5512f992810448b887bfb8d60487716
I20260926 13:41:43.187880 12852 tablet_service.cc:1514] Processing DeleteTablet for tablet d5512f992810448b887bfb8d60487716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:41:43 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:42108
I20260926 13:41:43.189466 12575 tablet_service.cc:1514] Processing DeleteTablet for tablet d5512f992810448b887bfb8d60487716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:41:43 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:52630
I20260926 13:41:43.190725 12729 tablet_service.cc:1514] Processing DeleteTablet for tablet d5512f992810448b887bfb8d60487716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:41:43 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:52672
I20260926 13:41:43.192483 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 12526
I20260926 13:41:43.193385 12971 tablet_replica.cc:330] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89: stopping tablet replica
I20260926 13:41:43.193810 12971 raft_consensus.cc:2229] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:41:43.194299 12971 raft_consensus.cc:2258] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:41:43.198139 12971 ts_tablet_manager.cc:1905] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:41:43.201579 12971 ts_tablet_manager.cc:1918] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:41:43.201720 12971 log.cc:1196] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/d5512f992810448b887bfb8d60487716
I20260926 13:41:43.202085 12971 ts_tablet_manager.cc:1939] T d5512f992810448b887bfb8d60487716 P 527764a19aea4b0098306a19b44bcc89: Deleting consensus metadata
I20260926 13:41:43.202142 12973 tablet_replica.cc:330] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8: stopping tablet replica
I20260926 13:41:43.202565 12973 raft_consensus.cc:2229] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:41:43.203022 12973 raft_consensus.cc:2258] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:41:43.204998 12973 ts_tablet_manager.cc:1905] T d5512f992810448b887bfb8d60487716 P 10d6c06c4d55443fb1848addde66c9e8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 13:41:43.205559 12430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:44407 (error 108)
I20260926 13:41:43.207365 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:43.207008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:44407 (local address 127.0.0.1:54224)
0926 13:41:43.207156 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 13:41:43.207262 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:44407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 13:41:43.207358 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 12665
W20260926 13:41:43.207540 12430 catalog_manager.cc:4584] TS 59e6077966b147258a3d768b0820a96d (127.1.49.1:44407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5512f992810448b887bfb8d60487716: Network error: Client connection negotiation failed: client connection to 127.1.49.1:44407: connect: Connection refused (error 111)
I20260926 13:41:43.209774 12432 catalog_manager.cc:4841] TS 527764a19aea4b0098306a19b44bcc89 (127.1.49.3:44275): tablet d5512f992810448b887bfb8d60487716 (table test-table [id=19ca67921c3f4d7dbef54c44ab0c70f8]) successfully deleted
I20260926 13:41:43.233299 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:41:43.232912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:46451 (local address 127.0.0.1:40488)
0926 13:41:43.233128 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 13:41:43.233190 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.2:46451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260926 13:41:43.235948 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 12800
I20260926 13:41:43.244740 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 12413
I20260926 13:41:43.253288 1220 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 13:41:43.600071 1220 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 13:41:43.600216 1220 mini_ranger.cc:66] Stopping Ranger...
I20260926 13:41:43.946774 1220 mini_ranger.cc:68] Stopped Ranger
2026-09-26 13:41:43.947 UTC [11999] LOG: received smart shutdown request
2026-09-26 13:41:43.951 UTC [11999] LOG: background worker "logical replication launcher" (PID 12008) exited with exit code 1
2026-09-26 13:41:43.951 UTC [12003] LOG: shutting down
2026-09-26 13:41:44.205 UTC [11999] LOG: database system is shut down
2026-09-26T13:41:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51727 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:41:44 dist-test-slave-1wns krb5kdc[12983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:41:44 dist-test-slave-1wns krb5kdc[12983](info): set up 2 sockets
Sep 26 13:41:44 dist-test-slave-1wns krb5kdc[12983](info): commencing operation
krb5kdc: starting...
W20260926 13:41:45.428194 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:41:45 dist-test-slave-1wns krb5kdc[12983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:41:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:41:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/rangeradmin_127.1.49.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/rangeradmin_127.1.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/rangerlookup_127.1.49.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/rangerlookup_127.1.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/HTTP_127.1.49.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/HTTP_127.1.49.20@KRBTEST.COM.keytab.
I20260926 13:41:45.636780 1220 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7nlTrX/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/postgres -l logfile start
2026-09-26 13:41:48.724 UTC [13013] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 13:41:48.724 UTC [13013] LOG: listening on IPv4 address "127.1.49.20", port 47777
2026-09-26 13:41:48.728 UTC [13013] LOG: listening on Unix socket "/tmp/.s.PGSQL.47777"
2026-09-26 13:41:48.754 UTC [13016] LOG: database system was shut down at 2026-09-26 13:41:46 UTC
2026-09-26 13:41:48.759 UTC [13013] LOG: database system is ready to accept connections
I20260926 13:41:49.751916 1220 mini_postgres.cc:96] Postgres bound to 47777
2026-09-26 13:41:49.759 UTC [13024] FATAL: database "slave" does not exist
127.1.49.20:47777 - accepting connections
I20260926 13:41:49.759989 1220 mini_ranger.cc:161] Starting Ranger...
I20260926 13:41:49.778012 1220 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 13:41:50.416378 1220 mini_ranger.cc:87] Created ranger Postgres database
I20260926 13:41:50.416493 1220 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-admin
2026-09-26 13:41:50,715 [I] DB FLAVOR :POSTGRES
2026-09-26 13:41:50,716 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:41:50,716 [I] Checking connection..
2026-09-26 13:41:50,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 13:41:51,004 [I] Checking connection passed.
2026-09-26 13:41:51,004 [I] --------- Verifying version history table ---------
2026-09-26 13:41:51,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:41:51,263 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:41:51,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:41:51,549 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:41:51,550 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 13:41:51,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 13:41:51.809 UTC [13088] WARNING: there is no transaction in progress
2026-09-26 13:41:51,816 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 13:41:51,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:41:52,118 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 13:41:52,118 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:41:52,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 13:41:52,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 13:41:52,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1wns.c.gcp-upstream.internal','N') ;"
2026-09-26 13:41:52,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:41:53,179 [I] Table x_portal_user does not exist in database ranger
2026-09-26 13:41:53,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:41:53,463 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 13:41:53,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:41:53,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:41:54,019 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 13:41:54,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 13:41:55.203 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.225 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.246 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.266 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.289 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.317 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.382 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.391 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.408 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.422 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.435 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.452 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.464 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.472 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.481 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:55.490 UTC [13196] WARNING: there is no transaction in progress
2026-09-26 13:41:56,416 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 13:41:56,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 13:41:56,667 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 13:41:56,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:41:56,969 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 13:41:56,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:41:57,232 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 13:41:57,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:41:57,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:41:57,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1wns.c.gcp-upstream.internal';"
2026-09-26 13:41:58,059 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 13:41:58,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 13:41:58,325 [I] DB_PATCHES have already been applied
I20260926 13:41:58.331169 1220 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-admin:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7nlTrX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 13:41:58.331254 1220 mini_ranger.cc:193] Using host: 127.1.49.20
I20260926 13:41:58.335206 1220 mini_ranger.cc:227] Ranger admin URL: http://127.1.49.20:45479
Sep 26, 2026 1:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 1:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 1:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 1:41:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 1:41:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 13:41:59.527155 1220 mini_ranger.cc:160] Time spent starting Ranger: real 9.767s user 0.000s sys 0.011s
Sep 26, 2026 1:42:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 1:42:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 1:42:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 1:42:22 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 1:42:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 1:42:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 1:42:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 1:42:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 1:42:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 1:42:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 1:42:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 1:42:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 1:42:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 1:42:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 1:42:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 1:42:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 1:42:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 1:42:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 13:42:23.563233 1220 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.49.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.49.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/rangerkms_127.1.49.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.49.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/rangerkms_127.1.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.49.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.49.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.49.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/HTTP_127.1.49.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.49.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/HTTP_127.1.49.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 13:42:23 dist-test-slave-1wns krb5kdc[12983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430143, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 13:42:23.675554 1220 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 13:42:23.687448 1220 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 13:42:24.801868 1220 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 13:42:24.802001 1220 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-kms
I20260926 13:42:24.802029 1220 mini_ranger_kms.cc:227] Using postgres at 127.1.49.20:47777
2026-09-26 13:42:25,022 [I] DB FLAVOR :POSTGRES
2026-09-26 13:42:25,022 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:42:25,022 [I] Checking connection
2026-09-26 13:42:25,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 13:42:25,308 [I] connection success
2026-09-26 13:42:25,309 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 13:42:25,309 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 13:42:25,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 13:42:25,624 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 13:42:25,625 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:42:25,625 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 13:42:25,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.49.20:47777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 13:42:25,963 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 13:42:26.497078 1220 mini_ranger_kms.cc:310] Created kms service
I20260926 13:42:26.702358 1220 mini_ranger_kms.cc:326] Created kudu user
I20260926 13:42:26.761541 1220 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 13:42:26 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430143, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:42:27.224006 1220 mini_ranger_kms.cc:384] Added ranger policy
I20260926 13:42:27.224175 1220 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-kms:/tmp/dist-test-task7nlTrX/build/debug/bin/postgresql.jar:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7nlTrX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 13:42:27.224202 1220 mini_ranger_kms.cc:240] Using host: 127.1.49.20
I20260926 13:42:27.228590 1220 mini_ranger_kms.cc:276] Ranger KMS PID: 13377
I20260926 13:42:27.228694 1220 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.49.20:49531
Sep 26, 2026 1:42:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:42:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:42:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 1:42:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 1:42:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 1:42:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 13:42:29.619376 1220 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.944s user 0.015s sys 0.175s
I20260926 13:42:29.619650 1220 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 13:42:29.619837 1220 mini_ranger_kms.cc:401] 127.1.49.20:49531/kms/v1/keys
Sep 26 13:42:29 dist-test-slave-1wns krb5kdc[12983](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790430149, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.49.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 13:42:30 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430149, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.49.20@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
Sep 26 13:42:30 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430149, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.49.20@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
Sep 26 13:42:33 dist-test-slave-1wns krb5kdc[12983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:42:33.496062 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:44433
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:44433
--ranger_config_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.49.20:49531/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:42:33.641592 13427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:33.641996 13427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:33.642089 13427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:33.647360 13427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:42:33.647473 13427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:33.647516 13427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:33.647583 13427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:42:33.647645 13427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:42:33.655076 13427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.49.20:49531/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:44433
--ranger_config_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:44433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:33.656476 13427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:33.657665 13427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:33.670630 13433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:33.670644 13432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:33.670663 13435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:33.671159 13427 server_base.cc:1034] running on GCE node
I20260926 13:42:33.671957 13427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:33.673497 13427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:33.674795 13427 hybrid_clock.cc:648] HybridClock initialized: now 1790430153674769 us; error 62 us; skew 500 ppm
Sep 26 13:42:33 dist-test-slave-1wns krb5kdc[12983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430153, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:33.678763 13427 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:42:33.680820 13427 webserver.cc:460] Webserver started at http://127.1.49.62:38555/ using document root <none> and password file <none>
I20260926 13:42:33.681643 13427 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:33.681725 13427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:33.682018 13427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:42:33 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430153, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:42:33.750813 13427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "26d87cb4daca462683b8834c15530a24"
format_stamp: "Formatted at 2026-09-26 13:42:33 on dist-test-slave-1wns"
server_key: "7a66421aebb3eda89bd7dc82f542d5c2"
server_key_iv: "e259e2a742a6e070ba25fde8b3f7d886"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:33.751749 13427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "26d87cb4daca462683b8834c15530a24"
format_stamp: "Formatted at 2026-09-26 13:42:33 on dist-test-slave-1wns"
server_key: "7a66421aebb3eda89bd7dc82f542d5c2"
server_key_iv: "e259e2a742a6e070ba25fde8b3f7d886"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:33.756465 13427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:42:33.775270 13442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:33.777047 13427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:42:33.777220 13427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "26d87cb4daca462683b8834c15530a24"
format_stamp: "Formatted at 2026-09-26 13:42:33 on dist-test-slave-1wns"
server_key: "7a66421aebb3eda89bd7dc82f542d5c2"
server_key_iv: "e259e2a742a6e070ba25fde8b3f7d886"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:33.777369 13427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:33.796310 13427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:33.800880 13427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:33.801429 13427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:33.812212 13427 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:44433
I20260926 13:42:33.812212 13494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:44433 every 8 connection(s)
I20260926 13:42:33.813352 13427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:42:33.817214 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 13427
I20260926 13:42:33.817406 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:42:33.817442 13495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:42:33.817785 1220 external_mini_cluster.cc:1428] Setting key 504c6830c199c782b1fdf6a8df68ffe8
I20260926 13:42:33.825836 13495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24: Bootstrap starting.
Sep 26 13:42:33 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:33.829025 13495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24: Neither blocks nor log segments found. Creating new log.
I20260926 13:42:33.830039 13495 log.cc:824] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:33.833087 13495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24: No bootstrap required, opened a new log
I20260926 13:42:33.836383 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:33.819730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.1.49.62:44433)
0926 13:42:33.820301 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:33.820330 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:33.820371 (+ 41us) server_negotiation.cc:407] Connection header received
0926 13:42:33.821351 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:33.821375 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:33.821766 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:33.822174 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:33.823075 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:33.824212 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:33.825152 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:33.825728 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:33.826067 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:33.829187 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:33.829196 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:33.829223 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:33.829270 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:33.832379 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:33.833054 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:33.833062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:33.833069 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:33.833295 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:33.833657 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:33.833661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:33.833662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:33.834257 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:33.834475 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:33.834829 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 13:42:33.835179 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":172,"threads_started":1}
I20260926 13:42:33.837852 13495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d87cb4daca462683b8834c15530a24" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44433 } }
I20260926 13:42:33.838193 13495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:42:33.838267 13495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d87cb4daca462683b8834c15530a24, State: Initialized, Role: FOLLOWER
I20260926 13:42:33.838822 13495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d87cb4daca462683b8834c15530a24" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44433 } }
I20260926 13:42:33.839053 13495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:42:33.839128 13495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:42:33.839270 13495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:33.840857 13495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d87cb4daca462683b8834c15530a24" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44433 } }
I20260926 13:42:33.841326 13495 leader_election.cc:302] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26d87cb4daca462683b8834c15530a24; no voters:
I20260926 13:42:33.841802 13495 leader_election.cc:288] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:42:33.841907 13500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:42:33.842125 13500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 1 LEADER]: Becoming Leader. State: Replica: 26d87cb4daca462683b8834c15530a24, State: Running, Role: LEADER
I20260926 13:42:33.842504 13500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d87cb4daca462683b8834c15530a24" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44433 } }
I20260926 13:42:33.843570 13495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:42:33.843930 13500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d87cb4daca462683b8834c15530a24. Latest consensus state: current_term: 1 leader_uuid: "26d87cb4daca462683b8834c15530a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d87cb4daca462683b8834c15530a24" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44433 } } }
I20260926 13:42:33.844055 13500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:33.844699 13501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26d87cb4daca462683b8834c15530a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d87cb4daca462683b8834c15530a24" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44433 } } }
I20260926 13:42:33.844913 13501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:33.846577 13504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:42:33.848683 13495 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 13:42:33.850523 13504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:42:33.868337 13504 catalog_manager.cc:1347] Generated new cluster ID: 48951856fc29408580eb4ffb31489ed6
I20260926 13:42:33.868439 13504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:42:33.926479 13504 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:42:33.928059 13504 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:42:33.950312 13504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24: Generated new TSK 0
I20260926 13:42:33.951480 13504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:42:34.664851 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.49.20:49531/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:44433
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:42:34.834967 13540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:34.835387 13540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:34.835454 13540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:34.841524 13540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:34.841646 13540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:34.841768 13540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:42:34.850683 13540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.49.20:49531/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:44433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:34.852315 13540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:34.853776 13540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:34.867424 13548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:34.867362 13547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:34.867586 13540 server_base.cc:1034] running on GCE node
W20260926 13:42:34.867364 13550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:34.868439 13540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:34.869196 13540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:34.870460 13540 hybrid_clock.cc:648] HybridClock initialized: now 1790430154870416 us; error 64 us; skew 500 ppm
Sep 26 13:42:34 dist-test-slave-1wns krb5kdc[12983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430154, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:34.874476 13540 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:42:34.876041 13540 webserver.cc:460] Webserver started at http://127.1.49.1:35539/ using document root <none> and password file <none>
I20260926 13:42:34.876965 13540 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:34.877146 13540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:34.877485 13540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:42:34 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430154, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:42:34.897437 13540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "e44e85a0cec947ed84501f2eec8f66b9"
format_stamp: "Formatted at 2026-09-26 13:42:34 on dist-test-slave-1wns"
server_key: "694149c6a5eda706b19c604ee97c31d8"
server_key_iv: "8363cbc8fd9d9be707adf4d14f2011d9"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:34.898398 13540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "e44e85a0cec947ed84501f2eec8f66b9"
format_stamp: "Formatted at 2026-09-26 13:42:34 on dist-test-slave-1wns"
server_key: "694149c6a5eda706b19c604ee97c31d8"
server_key_iv: "8363cbc8fd9d9be707adf4d14f2011d9"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:34.913803 13540 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.003s sys 0.004s
I20260926 13:42:34.930979 13558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:34.932574 13540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:42:34.932751 13540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "e44e85a0cec947ed84501f2eec8f66b9"
format_stamp: "Formatted at 2026-09-26 13:42:34 on dist-test-slave-1wns"
server_key: "694149c6a5eda706b19c604ee97c31d8"
server_key_iv: "8363cbc8fd9d9be707adf4d14f2011d9"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:34.932904 13540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:34.966827 13540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:34.971004 13540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:34.971306 13540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:34.972194 13540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:34.973378 13540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:42:34.973502 13540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:34.973625 13540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:42:34.973701 13540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:34.984925 13540 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:42799
I20260926 13:42:34.986475 13540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:42:34.988124 13672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:42799 every 8 connection(s)
I20260926 13:42:34.991250 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 13540
I20260926 13:42:34.991396 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:42:34.991705 1220 external_mini_cluster.cc:1428] Setting key 436b63ec8fc78d2c9bb64a64c3561bf2
Sep 26 13:42:34 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430154, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
I20260926 13:42:35.010404 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:34.991088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:52767 (local address 127.1.49.62:44433)
0926 13:42:34.992994 (+ 1906us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:34.992999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:34.994928 (+ 1929us) server_negotiation.cc:407] Connection header received
0926 13:42:34.994985 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:34.994988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:34.995055 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:34.995159 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:34.997136 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:34.998027 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:35.000301 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:35.000759 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:35.000849 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:35.004588 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:35.004593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:35.004596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:35.004632 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:35.006697 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:35.007638 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:35.007642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:35.007644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:35.007766 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:35.008323 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:35.008326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:35.008328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:35.008579 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:35.008681 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:35.010030 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0926 13:42:35.010204 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1809,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
I20260926 13:42:35.012516 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:34.991671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:44433 (local address 127.1.49.1:52767)
0926 13:42:34.993244 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:34.993317 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:34.994549 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:34.995543 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:34.995555 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:34.996059 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:34.996909 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:34.996934 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:34.998277 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:34.998281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:35.000097 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:35.000116 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:35.001113 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:35.001117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:35.001362 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:35.002198 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:35.002223 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:35.004267 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:35.006917 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:35.006927 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:35.006951 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:35.007437 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:35.007967 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:35.007970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:35.007972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:35.008196 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:35.008760 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:35.008768 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:35.009196 (+ 428us) client_negotiation.cc:769] Sending connection context
0926 13:42:35.009577 (+ 381us) client_negotiation.cc:240] Negotiation successful
0926 13:42:35.011175 (+ 1598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":168,"threads_started":1}
I20260926 13:42:35.014928 13673 heartbeater.cc:347] Connected to a master server at 127.1.49.62:44433
I20260926 13:42:35.015234 13673 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:35.016100 13673 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
I20260926 13:42:35.018455 13459 ts_manager.cc:194] Registered new tserver with Master: e44e85a0cec947ed84501f2eec8f66b9 (127.1.49.1:42799)
I20260926 13:42:35.020129 13459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:52767
Sep 26 13:42:35 dist-test-slave-1wns krb5kdc[12983](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790430155, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:42:35.037827 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.49.20:49531/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:44433
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:42:35.218478 13679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:35.218904 13679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:35.218971 13679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:35.225749 13679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:35.225857 13679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:35.225981 13679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:42:35.234977 13679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.49.20:49531/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:44433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:35.236629 13679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:35.238093 13679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:35.253330 13686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:35.253856 13688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:35.254145 13685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:35.255650 13679 server_base.cc:1034] running on GCE node
I20260926 13:42:35.256611 13679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:35.257567 13679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:35.260912 13679 hybrid_clock.cc:648] HybridClock initialized: now 1790430155260812 us; error 100 us; skew 500 ppm
Sep 26 13:42:35 dist-test-slave-1wns krb5kdc[12983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430155, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:35.265789 13679 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:42:35.267488 13679 webserver.cc:460] Webserver started at http://127.1.49.2:40805/ using document root <none> and password file <none>
I20260926 13:42:35.268378 13679 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:35.268541 13679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:35.268859 13679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:42:35 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430155, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:42:35.287539 13679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "3f3e619b47bc4a8ea3290010dc207996"
format_stamp: "Formatted at 2026-09-26 13:42:35 on dist-test-slave-1wns"
server_key: "90d40e9171b59db376055901a20c586e"
server_key_iv: "ee96ed4bb553aae50e98c647868835d6"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:35.288506 13679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "3f3e619b47bc4a8ea3290010dc207996"
format_stamp: "Formatted at 2026-09-26 13:42:35 on dist-test-slave-1wns"
server_key: "90d40e9171b59db376055901a20c586e"
server_key_iv: "ee96ed4bb553aae50e98c647868835d6"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:35.294553 13679 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 13:42:35.316123 13695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:35.318073 13679 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260926 13:42:35.318267 13679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "3f3e619b47bc4a8ea3290010dc207996"
format_stamp: "Formatted at 2026-09-26 13:42:35 on dist-test-slave-1wns"
server_key: "90d40e9171b59db376055901a20c586e"
server_key_iv: "ee96ed4bb553aae50e98c647868835d6"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:35.318439 13679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:35.358753 13679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:35.362725 13679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:35.363083 13679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:35.364035 13679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:35.365413 13679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:42:35.365499 13679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:35.365605 13679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:42:35.365657 13679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:35.377601 13679 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:43961
I20260926 13:42:35.378311 13808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:43961 every 8 connection(s)
I20260926 13:42:35.378952 13679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:42:35.383081 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 13679
I20260926 13:42:35.383221 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:42:35.383613 1220 external_mini_cluster.cc:1428] Setting key bafe24bb5b9fb7995c2f732b88267244
Sep 26 13:42:35 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430155, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
I20260926 13:42:35.404562 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:35.381579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:39485 (local address 127.1.49.62:44433)
0926 13:42:35.381769 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:35.381775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:35.383048 (+ 1273us) server_negotiation.cc:407] Connection header received
0926 13:42:35.385258 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:35.385262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:35.385319 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:35.385491 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:35.387512 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:35.388467 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:35.390589 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:35.391044 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:35.391148 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:35.396672 (+ 5524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:35.396679 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:35.396683 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:35.396728 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:35.398897 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:35.402261 (+ 3364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:35.402266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:35.402268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:35.402410 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:35.402953 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:35.402956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:35.402958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:35.403224 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:35.403321 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:35.404197 (+ 876us) server_negotiation.cc:299] Negotiation successful
0926 13:42:35.404376 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 13:42:35.406780 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:35.382126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:44433 (local address 127.1.49.2:39485)
0926 13:42:35.382684 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:35.382924 (+ 240us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:35.384983 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:35.386033 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:35.386046 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:35.386456 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:35.387265 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:35.387293 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:35.388680 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:35.388683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:35.390393 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:35.390413 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:35.391409 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:35.391416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:35.391689 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:35.393459 (+ 1770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:35.393487 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:35.396341 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:35.399135 (+ 2794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:35.399144 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:35.399168 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:35.402052 (+ 2884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:35.402562 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:35.402565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:35.402567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:35.402818 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:35.403409 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:35.403418 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:35.403874 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 13:42:35.405119 (+ 1245us) client_negotiation.cc:240] Negotiation successful
0926 13:42:35.405547 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":155,"threads_started":1}
I20260926 13:42:35.408649 13809 heartbeater.cc:347] Connected to a master server at 127.1.49.62:44433
I20260926 13:42:35.408929 13809 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:35.409596 13809 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
I20260926 13:42:35.411113 13459 ts_manager.cc:194] Registered new tserver with Master: 3f3e619b47bc4a8ea3290010dc207996 (127.1.49.2:43961)
I20260926 13:42:35.412128 13459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:39485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:42:35.424417 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.49.20:49531/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:44433
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:42:35 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430155, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
W20260926 13:42:35.604270 13814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:35.604768 13814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:35.604888 13814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:35.611642 13814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:35.611871 13814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:35.612020 13814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:42:35.620986 13814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.49.20:49531/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:44433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:35.623409 13814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:35.625084 13814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:35.639966 13822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:35.640527 13820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:35.641675 13819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:35.642094 13814 server_base.cc:1034] running on GCE node
I20260926 13:42:35.642791 13814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:35.643697 13814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:35.645078 13814 hybrid_clock.cc:648] HybridClock initialized: now 1790430155645000 us; error 96 us; skew 500 ppm
Sep 26 13:42:35 dist-test-slave-1wns krb5kdc[12983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430155, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:35.649041 13814 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:42:35.650328 13814 webserver.cc:460] Webserver started at http://127.1.49.3:37293/ using document root <none> and password file <none>
I20260926 13:42:35.651108 13814 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:35.651216 13814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:35.651510 13814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:42:35 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430155, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:42:35.667863 13814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3"
format_stamp: "Formatted at 2026-09-26 13:42:35 on dist-test-slave-1wns"
server_key: "3a53fb2171ed4fcd1a0a1c3dd17666b4"
server_key_iv: "bc066846e7bf503e2f025a3a696f5720"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:35.668968 13814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3"
format_stamp: "Formatted at 2026-09-26 13:42:35 on dist-test-slave-1wns"
server_key: "3a53fb2171ed4fcd1a0a1c3dd17666b4"
server_key_iv: "bc066846e7bf503e2f025a3a696f5720"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:35.678081 13814 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.000s
I20260926 13:42:35.704120 13829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:35.706091 13814 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260926 13:42:35.706254 13814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3"
format_stamp: "Formatted at 2026-09-26 13:42:35 on dist-test-slave-1wns"
server_key: "3a53fb2171ed4fcd1a0a1c3dd17666b4"
server_key_iv: "bc066846e7bf503e2f025a3a696f5720"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:35.706416 13814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:35.724852 13814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:35.729329 13814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:35.729558 13814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:35.730351 13814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:35.731472 13814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:42:35.731541 13814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:35.731600 13814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:42:35.731637 13814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:35.743866 13814 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:39419
I20260926 13:42:35.744139 13942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:39419 every 8 connection(s)
I20260926 13:42:35.745409 13814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:42:35.754711 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 13814
I20260926 13:42:35.754854 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:42:35.755184 1220 external_mini_cluster.cc:1428] Setting key 1079d10b5bc765e730203617fb5c4c9e
Sep 26 13:42:35 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430155, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:35.765120 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:35.748048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:47473 (local address 127.1.49.62:44433)
0926 13:42:35.748186 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:35.748191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:35.749558 (+ 1367us) server_negotiation.cc:407] Connection header received
0926 13:42:35.750906 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:35.750910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:35.750976 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:35.751085 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:35.752694 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:35.753643 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:35.755704 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:35.756155 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:35.756244 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:35.759642 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:35.759650 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:35.759654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:35.759709 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:35.761777 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:35.762585 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:35.762590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:35.762592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:35.762698 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:35.763480 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:35.763484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:35.763486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:35.763761 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:35.763914 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:35.764785 (+ 871us) server_negotiation.cc:299] Negotiation successful
0926 13:42:35.764942 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:42:35.766690 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:35.748668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:44433 (local address 127.1.49.3:47473)
0926 13:42:35.749356 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:35.749425 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:35.750589 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:35.751322 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:35.751331 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:35.751729 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:35.752483 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:35.752505 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:35.753865 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:35.753869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:35.755459 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:35.755473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:35.756478 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:35.756483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:35.756703 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:35.757480 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:35.757502 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:35.759381 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:35.761953 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:35.761963 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:35.761985 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:35.762449 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:35.762976 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:35.762979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:35.762981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:35.763332 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:35.763937 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:35.763947 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:35.764403 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 13:42:35.764702 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 13:42:35.765144 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":202,"threads_started":1}
I20260926 13:42:35.769227 13943 heartbeater.cc:347] Connected to a master server at 127.1.49.62:44433
I20260926 13:42:35.769555 13943 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:35.770251 13943 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
I20260926 13:42:35.771649 13459 ts_manager.cc:194] Registered new tserver with Master: f8fb6858c3f24d35a9474fbd3cfab3d3 (127.1.49.3:39419)
I20260926 13:42:35.772477 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:42:35.772519 13459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:47473
I20260926 13:42:35.773245 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 13540
I20260926 13:42:35.781057 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 13679
I20260926 13:42:35.789130 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 13814
I20260926 13:42:35.797168 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 13427
I20260926 13:42:35.805692 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:44433
--webserver_interface=127.1.49.62
--webserver_port=38555
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:44433
--ranger_config_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.49.20:49531/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.49.20:49531/kms with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:42:35.928203 13946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:35.928539 13946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:35.928581 13946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:35.932967 13946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:42:35.933048 13946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:35.933068 13946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:35.933085 13946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:42:35.933101 13946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:42:35.939134 13946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.49.20:49531/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:44433
--ranger_config_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:44433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=38555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:35.940460 13946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:35.941624 13946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:35.951720 13952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:35.951766 13951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:35.951740 13946 server_base.cc:1034] running on GCE node
W20260926 13:42:35.951768 13954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:35.952868 13946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:35.954231 13946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:35.955541 13946 hybrid_clock.cc:648] HybridClock initialized: now 1790430155955509 us; error 52 us; skew 500 ppm
Sep 26 13:42:35 dist-test-slave-1wns krb5kdc[12983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430155, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:35.959125 13946 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:42:35.960388 13946 webserver.cc:460] Webserver started at http://127.1.49.62:38555/ using document root <none> and password file <none>
I20260926 13:42:35.961002 13946 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:35.961086 13946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:42:36 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430155, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:42:36.041162 13946 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:42:36.043973 13962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:36.045706 13946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:42:36.045905 13946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "26d87cb4daca462683b8834c15530a24"
format_stamp: "Formatted at 2026-09-26 13:42:33 on dist-test-slave-1wns"
server_key: "7a66421aebb3eda89bd7dc82f542d5c2"
server_key_iv: "e259e2a742a6e070ba25fde8b3f7d886"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:36.046576 13946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:36.059242 13946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:36.063056 13946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:36.063347 13946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:36.072593 13946 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:44433
I20260926 13:42:36.072644 14014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:44433 every 8 connection(s)
I20260926 13:42:36.073866 13946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:42:36.075634 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 13946
I20260926 13:42:36.076202 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.49.20:49531/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:42799
--local_ip_for_outbound_sockets=127.1.49.1
--tserver_master_addrs=127.1.49.62:44433
--webserver_port=35539
--webserver_interface=127.1.49.1
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.49.20:49531/kms with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:42:36.079207 14015 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:42:36.081670 14015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24: Bootstrap starting.
I20260926 13:42:36.101097 14015 log.cc:824] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:36.109253 14015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:42:36.109736 14015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24: Bootstrap complete.
I20260926 13:42:36.114310 14015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d87cb4daca462683b8834c15530a24" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44433 } }
I20260926 13:42:36.114843 14015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d87cb4daca462683b8834c15530a24, State: Initialized, Role: FOLLOWER
I20260926 13:42:36.115468 14015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d87cb4daca462683b8834c15530a24" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44433 } }
I20260926 13:42:36.115636 14015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:42:36.115727 14015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:42:36.115881 14015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:42:36.118201 14015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d87cb4daca462683b8834c15530a24" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44433 } }
I20260926 13:42:36.118635 14015 leader_election.cc:302] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26d87cb4daca462683b8834c15530a24; no voters:
I20260926 13:42:36.119231 14015 leader_election.cc:288] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:42:36.119382 14019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:42:36.119655 14019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [term 2 LEADER]: Becoming Leader. State: Replica: 26d87cb4daca462683b8834c15530a24, State: Running, Role: LEADER
I20260926 13:42:36.120134 14019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d87cb4daca462683b8834c15530a24" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44433 } }
I20260926 13:42:36.120895 14015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:42:36.121702 14020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d87cb4daca462683b8834c15530a24. Latest consensus state: current_term: 2 leader_uuid: "26d87cb4daca462683b8834c15530a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d87cb4daca462683b8834c15530a24" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44433 } } }
I20260926 13:42:36.121840 14020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:36.122150 14021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "26d87cb4daca462683b8834c15530a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d87cb4daca462683b8834c15530a24" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44433 } } }
I20260926 13:42:36.122236 14021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24 [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:36.122543 14027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:42:36.126142 14027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:42:36.126750 14015 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 13:42:36.127094 14027 catalog_manager.cc:1259] Loaded cluster ID: 48951856fc29408580eb4ffb31489ed6
I20260926 13:42:36.127251 14027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:42:36.129954 14027 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:42:36.130697 14027 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24: Loaded TSK: 0
I20260926 13:42:36.132076 14027 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:42:36.218075 14017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:36.218611 14017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:36.218755 14017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:36.226123 14017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:36.226221 14017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:36.226325 14017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:42:36.235500 14017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.49.20:49531/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:42799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=35539
--tserver_master_addrs=127.1.49.62:44433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:36.237073 14017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:36.238499 14017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:36.253151 14054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:36.253033 14056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:36.253654 14053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:36.253716 14017 server_base.cc:1034] running on GCE node
I20260926 13:42:36.254277 14017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:36.255173 14017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:36.256388 14017 hybrid_clock.cc:648] HybridClock initialized: now 1790430156256362 us; error 42 us; skew 500 ppm
Sep 26 13:42:36 dist-test-slave-1wns krb5kdc[12983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:36.260435 14017 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:42:36.262081 14017 webserver.cc:460] Webserver started at http://127.1.49.1:35539/ using document root <none> and password file <none>
I20260926 13:42:36.262974 14017 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:36.263103 14017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:42:36 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:42:36.293741 14017 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:42:36.297966 14064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:36.300030 14017 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:42:36.300307 14017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "e44e85a0cec947ed84501f2eec8f66b9"
format_stamp: "Formatted at 2026-09-26 13:42:34 on dist-test-slave-1wns"
server_key: "694149c6a5eda706b19c604ee97c31d8"
server_key_iv: "8363cbc8fd9d9be707adf4d14f2011d9"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:36.301146 14017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:36.319355 14017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:36.323482 14017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:36.323693 14017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:36.324457 14017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:36.325491 14017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:42:36.325548 14017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:36.325600 14017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:42:36.325662 14017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 13:42:36.336992 14017 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:42799
I20260926 13:42:36.337687 14177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:42799 every 8 connection(s)
I20260926 13:42:36.338408 14017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:42:36.338644 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 14017
I20260926 13:42:36.339118 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.49.20:49531/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:43961
--local_ip_for_outbound_sockets=127.1.49.2
--tserver_master_addrs=127.1.49.62:44433
--webserver_port=40805
--webserver_interface=127.1.49.2
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.49.20:49531/kms with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:42:36 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:36.388355 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:36.344622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:60697 (local address 127.1.49.62:44433)
0926 13:42:36.348167 (+ 3545us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:36.348201 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:36.349894 (+ 1693us) server_negotiation.cc:407] Connection header received
0926 13:42:36.351125 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:36.351146 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:36.351789 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:36.352337 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:36.354655 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:36.356540 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:36.358667 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:36.359585 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:36.359838 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:36.368750 (+ 8912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:36.368761 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:36.368783 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:36.368832 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:36.373733 (+ 4901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:36.384080 (+ 10347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:36.384091 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:36.384099 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:36.384394 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:36.385041 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:36.385045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:36.385047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:36.385647 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:36.385788 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:36.386749 (+ 961us) server_negotiation.cc:299] Negotiation successful
0926 13:42:36.387160 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":153,"threads_started":1}
I20260926 13:42:36.391010 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:36.341133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:44433 (local address 127.1.49.1:60697)
0926 13:42:36.348616 (+ 7483us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:36.348707 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:36.350212 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:36.353098 (+ 2886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:36.353110 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:36.353555 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:36.354377 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:36.354405 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:36.356779 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:36.356784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:36.358467 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:36.358483 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:36.360302 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:36.360308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:36.361592 (+ 1284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:36.363462 (+ 1870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:36.363488 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:36.368366 (+ 4878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:36.373985 (+ 5619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:36.373993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:36.374040 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:36.379673 (+ 5633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:36.384604 (+ 4931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:36.384608 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:36.384611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:36.384871 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:36.385929 (+ 1058us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:36.385937 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:36.386332 (+ 395us) client_negotiation.cc:769] Sending connection context
0926 13:42:36.389479 (+ 3147us) client_negotiation.cc:240] Negotiation successful
0926 13:42:36.389915 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7176,"thread_start_us":141,"threads_started":1}
I20260926 13:42:36.396135 14178 heartbeater.cc:347] Connected to a master server at 127.1.49.62:44433
I20260926 13:42:36.396446 14178 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:36.397105 14178 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
W20260926 13:42:36.399791 14178 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:44433 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 13:42:36.400012 14178 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:36.400262 14178 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
I20260926 13:42:36.401392 14178 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:36.401618 14178 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
W20260926 13:42:36.402598 14178 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:42:36.503607 14180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:36.504055 14180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:36.504177 14180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:36.510671 14180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:36.510779 14180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:36.510879 14180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:42:36.518824 14180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.49.20:49531/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:43961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=40805
--tserver_master_addrs=127.1.49.62:44433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:36.520110 14180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:36.521281 14180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:36.532833 14190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:36.532783 14187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:36.533354 14180 server_base.cc:1034] running on GCE node
W20260926 13:42:36.533376 14188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:36.533981 14180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:36.534801 14180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:36.535993 14180 hybrid_clock.cc:648] HybridClock initialized: now 1790430156535964 us; error 43 us; skew 500 ppm
Sep 26 13:42:36 dist-test-slave-1wns krb5kdc[12983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:36.539701 14180 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:42:36.540846 14180 webserver.cc:460] Webserver started at http://127.1.49.2:40805/ using document root <none> and password file <none>
I20260926 13:42:36.541455 14180 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:36.541532 14180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:42:36 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:42:36.566004 14180 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 13:42:36.569921 14198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:36.571647 14180 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:42:36.571898 14180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "3f3e619b47bc4a8ea3290010dc207996"
format_stamp: "Formatted at 2026-09-26 13:42:35 on dist-test-slave-1wns"
server_key: "90d40e9171b59db376055901a20c586e"
server_key_iv: "ee96ed4bb553aae50e98c647868835d6"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:36.572558 14180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:36.595302 14180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:36.600340 14180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:36.600591 14180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:36.601295 14180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:36.602283 14180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:42:36.602355 14180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:36.602423 14180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:42:36.602456 14180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:36.614058 14180 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:43961
I20260926 13:42:36.615228 14180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:42:36.616118 14311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:43961 every 8 connection(s)
I20260926 13:42:36.620000 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 14180
I20260926 13:42:36.620512 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.49.20:49531/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:39419
--local_ip_for_outbound_sockets=127.1.49.3
--tserver_master_addrs=127.1.49.62:44433
--webserver_port=37293
--webserver_interface=127.1.49.3
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.49.20:49531/kms with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:42:36 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:36.636103 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:36.619095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:56041 (local address 127.1.49.62:44433)
0926 13:42:36.619262 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:36.619282 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:36.619305 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:42:36.619933 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:36.619937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:36.620004 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:36.620116 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:36.622278 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:36.623347 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:36.625585 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:36.626015 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:36.626160 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:36.630746 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:36.630752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:36.630755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:36.630797 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:36.633076 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:36.633874 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:36.633879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:36.633881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:36.634003 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:36.634485 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:36.634489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:36.634492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:36.634778 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:36.634911 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:36.635693 (+ 782us) server_negotiation.cc:299] Negotiation successful
0926 13:42:36.635906 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 13:42:36.637656 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:36.618013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:44433 (local address 127.1.49.2:56041)
0926 13:42:36.618541 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:36.618600 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:36.619588 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:36.620407 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:36.620421 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:36.621122 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:36.622025 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:36.622057 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:36.623697 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:36.623703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:36.625381 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:36.625399 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:36.626267 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:36.626274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:36.626556 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:36.627390 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:36.627422 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:36.630434 (+ 3012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:36.633269 (+ 2835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:36.633278 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:36.633301 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:36.633709 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:36.634120 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:36.634123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:36.634125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:36.634354 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:36.634916 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:36.634924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:36.635327 (+ 403us) client_negotiation.cc:769] Sending connection context
0926 13:42:36.635833 (+ 506us) client_negotiation.cc:240] Negotiation successful
0926 13:42:36.636519 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":157,"threads_started":1}
I20260926 13:42:36.639196 14312 heartbeater.cc:347] Connected to a master server at 127.1.49.62:44433
I20260926 13:42:36.639526 14312 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:36.640304 14312 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
W20260926 13:42:36.642377 14312 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:44433 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 13:42:36.642534 14312 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:36.642767 14312 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
I20260926 13:42:36.643792 14312 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:36.644044 14312 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
W20260926 13:42:36.644927 14312 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:42:36.763171 14315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:36.763633 14315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:36.763792 14315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:36.769447 14315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:36.769615 14315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:36.769761 14315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:42:36.778028 14315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.49.20:49531/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:39419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=37293
--tserver_master_addrs=127.1.49.62:44433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:36.779529 14315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:36.780766 14315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:36.792564 14324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:36.792587 14322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:36.793623 14321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:36.794642 14315 server_base.cc:1034] running on GCE node
I20260926 13:42:36.795125 14315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:36.796056 14315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:36.799382 14315 hybrid_clock.cc:648] HybridClock initialized: now 1790430156799205 us; error 180 us; skew 500 ppm
Sep 26 13:42:36 dist-test-slave-1wns krb5kdc[12983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:36.802565 14315 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:42:36.803687 14315 webserver.cc:460] Webserver started at http://127.1.49.3:37293/ using document root <none> and password file <none>
I20260926 13:42:36.804333 14315 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:36.804415 14315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:42:36 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:42:36.829588 14315 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260926 13:42:36.832913 14332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:36.834694 14315 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:42:36.834861 14315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3"
format_stamp: "Formatted at 2026-09-26 13:42:35 on dist-test-slave-1wns"
server_key: "3a53fb2171ed4fcd1a0a1c3dd17666b4"
server_key_iv: "bc066846e7bf503e2f025a3a696f5720"
server_key_version: "kuduclusterkey@0"
I20260926 13:42:36.835662 14315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:36.860599 14315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:36.864255 14315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:36.864481 14315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:36.865157 14315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:36.866135 14315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:42:36.866212 14315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:36.866276 14315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:42:36.866330 14315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:36.877779 14315 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:39419
I20260926 13:42:36.878952 14315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:42:36.880419 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 14315
I20260926 13:42:36.879541 14445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:39419 every 8 connection(s)
Sep 26 13:42:36 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:36.896739 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:36.881212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.1.49.62:44433)
0926 13:42:36.881482 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:36.881487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:36.883487 (+ 2000us) server_negotiation.cc:407] Connection header received
0926 13:42:36.883635 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:36.883638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:36.883705 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:36.883803 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:36.884779 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:36.885675 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:36.887096 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:36.887480 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:36.887603 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:36.891034 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:36.891038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:36.891041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:36.891072 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:36.893189 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:36.894123 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:36.894127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:36.894130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:36.894242 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:36.894944 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:36.894948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:36.894950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:36.895246 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:36.895770 (+ 524us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:36.896371 (+ 601us) server_negotiation.cc:299] Negotiation successful
0926 13:42:36.896542 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260926 13:42:36.900605 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:36.881279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:45685 (local address 127.1.49.62:44433)
0926 13:42:36.881581 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:36.881584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:36.883298 (+ 1714us) server_negotiation.cc:407] Connection header received
0926 13:42:36.884106 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:36.884110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:36.884164 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:36.884230 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:36.887231 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:36.888149 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:36.890255 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:36.890687 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:36.890792 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:36.895039 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:36.895043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:36.895047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:36.895078 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:36.897369 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:36.898470 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:36.898475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:36.898477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:36.898583 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:36.899091 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:36.899095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:36.899097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:36.899348 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:36.899429 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:36.900306 (+ 877us) server_negotiation.cc:299] Negotiation successful
0926 13:42:36.900462 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":86,"threads_started":1}
I20260926 13:42:36.902083 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:36.881795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:44433 (local address 127.1.49.3:45685)
0926 13:42:36.882272 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:36.882325 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:36.883446 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:36.884705 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:36.884718 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:36.886275 (+ 1557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:36.887044 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:36.887072 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:36.888338 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:36.888342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:36.890065 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:36.890081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:36.891049 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:36.891053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:36.891304 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:36.892015 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:36.892040 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:36.894523 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:36.897867 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:36.897876 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:36.897900 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:36.898330 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:36.898714 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:36.898718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:36.898720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:36.898963 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:36.899522 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:36.899530 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:36.899976 (+ 446us) client_negotiation.cc:769] Sending connection context
0926 13:42:36.900836 (+ 860us) client_negotiation.cc:240] Negotiation successful
0926 13:42:36.901202 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":134,"threads_started":1}
I20260926 13:42:36.903713 14446 heartbeater.cc:347] Connected to a master server at 127.1.49.62:44433
I20260926 13:42:36.904055 14446 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:36.904729 14446 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
W20260926 13:42:36.906564 14446 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:44433 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 13:42:36.906718 14446 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:36.906932 14446 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
I20260926 13:42:36.907967 14446 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:36.908224 14446 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
W20260926 13:42:36.909168 14446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 13:42:37.402943 14178 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:37.403304 14178 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
I20260926 13:42:37.404892 13979 ts_manager.cc:194] Registered new tserver with Master: e44e85a0cec947ed84501f2eec8f66b9 (127.1.49.1:42799)
I20260926 13:42:37.406831 13979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:60697
Sep 26 13:42:37 dist-test-slave-1wns krb5kdc[12983](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790430157, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:37.645213 14312 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:37.645563 14312 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
I20260926 13:42:37.646461 13978 ts_manager.cc:194] Registered new tserver with Master: 3f3e619b47bc4a8ea3290010dc207996 (127.1.49.2:43961)
I20260926 13:42:37.647320 13978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:56041
Sep 26 13:42:37 dist-test-slave-1wns krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430157, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for HTTP/127.1.49.20@KRBTEST.COM
I20260926 13:42:37.909449 14446 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:37.909857 14446 heartbeater.cc:511] Master 127.1.49.62:44433 requested a full tablet report, sending...
I20260926 13:42:37.911691 13979 ts_manager.cc:194] Registered new tserver with Master: f8fb6858c3f24d35a9474fbd3cfab3d3 (127.1.49.3:39419)
I20260926 13:42:37.912454 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:42:37.912552 13979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:45685
I20260926 13:42:37.927635 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:37.915439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.1.49.62:44433)
0926 13:42:37.915743 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:37.915748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:37.916085 (+ 337us) server_negotiation.cc:407] Connection header received
0926 13:42:37.916345 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:37.916350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:37.916413 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:37.916485 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:37.917578 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:37.918616 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:37.919848 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:37.920145 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:37.920286 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:37.922557 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:37.922563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:37.922566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:37.922607 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:37.924847 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:37.925520 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:37.925524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:37.925527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:37.925640 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:37.926036 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:37.926039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:37.926041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:37.926300 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:37.926435 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:37.927258 (+ 823us) server_negotiation.cc:299] Negotiation successful
0926 13:42:37.927412 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":110,"threads_started":1}
I20260926 13:42:37.937659 13979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:42:37.944164 13979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:42:37.980736 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:37.963521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:42799 (local address 127.0.0.1:58956)
0926 13:42:37.967025 (+ 3504us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:37.967095 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:37.967257 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:37.968516 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:37.968523 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:37.968541 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:42:37.968692 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:37.968700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:37.970553 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:37.970558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:37.978210 (+ 7652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:37.978223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:37.979982 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:37.979986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:37.980148 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:37.980215 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 13:42:37.980357 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 13:42:37.980523 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3298,"thread_start_us":1199,"threads_started":1}
I20260926 13:42:37.985216 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:37.966062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.1.49.2:43961)
0926 13:42:37.966469 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:37.966477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:37.969028 (+ 2551us) server_negotiation.cc:407] Connection header received
0926 13:42:37.972161 (+ 3133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:37.972170 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:37.972467 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:37.972620 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:42:37.973134 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:37.974330 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:37.982929 (+ 8599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:37.983907 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:37.984004 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:37.984178 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:37.984956 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 13:42:37.985078 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":100,"threads_started":1}
I20260926 13:42:37.986434 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:37.963610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58956 (local address 127.1.49.1:42799)
0926 13:42:37.965102 (+ 1492us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:37.965110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:37.967427 (+ 2317us) server_negotiation.cc:407] Connection header received
0926 13:42:37.967534 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:37.967540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:37.967837 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:37.967998 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:42:37.969108 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:37.970399 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:37.978398 (+ 7999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:37.979551 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:37.979682 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:37.979898 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:37.986127 (+ 6229us) server_negotiation.cc:299] Negotiation successful
0926 13:42:37.986284 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1366,"thread_start_us":96,"threads_started":1}
I20260926 13:42:37.984810 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:37.965666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:43961 (local address 127.0.0.1:46858)
0926 13:42:37.968926 (+ 3260us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:37.968940 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:37.971936 (+ 2996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:37.972859 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:37.972862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:37.972884 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:42:37.972987 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:37.972994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:37.974489 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:37.974492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:37.976103 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:37.976119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:37.984317 (+ 8198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:37.984321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:37.984434 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:37.984459 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 13:42:37.984535 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 13:42:37.984629 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3191,"thread_start_us":920,"threads_started":1}
I20260926 13:42:37.990350 14246 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb8955c23d14956a5703ecb93e782c0 (DEFAULT_TABLE table=test-table [id=ee9a3234a96b40649ee959fdaceca4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:42:37.991261 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:37.964065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:39419 (local address 127.0.0.1:39172)
0926 13:42:37.968295 (+ 4230us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:37.968310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:37.968399 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:37.977445 (+ 9046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:37.977449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:37.977476 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:42:37.977596 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:37.977604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:37.988078 (+ 10474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:37.988082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:37.989240 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:37.989265 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:37.990668 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:37.990672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:37.990773 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:37.990804 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 13:42:37.990871 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:42:37.990966 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4159,"thread_start_us":953,"threads_started":1}
I20260926 13:42:37.992007 14246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb8955c23d14956a5703ecb93e782c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:42:37.991707 14106 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb8955c23d14956a5703ecb93e782c0 (DEFAULT_TABLE table=test-table [id=ee9a3234a96b40649ee959fdaceca4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:42:37.992524 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:37.964382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.1.49.3:39419)
0926 13:42:37.976515 (+ 12133us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:37.976524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:37.976555 (+ 31us) server_negotiation.cc:407] Connection header received
0926 13:42:37.976656 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:37.976663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:37.976980 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:37.977185 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:42:37.981169 (+ 3984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:37.982734 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:37.989447 (+ 6713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:37.990515 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:37.991966 (+ 1451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:37.992177 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:37.992243 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 13:42:37.992370 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11973,"thread_start_us":90,"threads_started":1}
I20260926 13:42:37.993111 14106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb8955c23d14956a5703ecb93e782c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:42:37.995959 14377 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb8955c23d14956a5703ecb93e782c0 (DEFAULT_TABLE table=test-table [id=ee9a3234a96b40649ee959fdaceca4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:42:37.997216 14377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb8955c23d14956a5703ecb93e782c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:42:38.004809 14480 tablet_bootstrap.cc:492] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996: Bootstrap starting.
I20260926 13:42:38.008002 14480 tablet_bootstrap.cc:654] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996: Neither blocks nor log segments found. Creating new log.
I20260926 13:42:38.019763 14481 tablet_bootstrap.cc:492] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9: Bootstrap starting.
I20260926 13:42:38.021828 14480 log.cc:824] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:38.022739 14481 tablet_bootstrap.cc:654] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9: Neither blocks nor log segments found. Creating new log.
I20260926 13:42:38.023844 14481 log.cc:824] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:38.025447 14480 tablet_bootstrap.cc:492] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996: No bootstrap required, opened a new log
I20260926 13:42:38.025897 14480 ts_tablet_manager.cc:1397] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20260926 13:42:38.032027 14482 tablet_bootstrap.cc:492] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3: Bootstrap starting.
I20260926 13:42:38.032166 14480 raft_consensus.cc:348] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44e85a0cec947ed84501f2eec8f66b9" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42799 } } peers { permanent_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39419 } } peers { permanent_uuid: "3f3e619b47bc4a8ea3290010dc207996" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43961 } }
I20260926 13:42:38.032542 14480 raft_consensus.cc:374] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:42:38.032639 14480 raft_consensus.cc:729] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3e619b47bc4a8ea3290010dc207996, State: Initialized, Role: FOLLOWER
I20260926 13:42:38.033262 14480 consensus_queue.cc:260] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44e85a0cec947ed84501f2eec8f66b9" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42799 } } peers { permanent_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39419 } } peers { permanent_uuid: "3f3e619b47bc4a8ea3290010dc207996" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43961 } }
I20260926 13:42:38.034837 14480 ts_tablet_manager.cc:1428] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260926 13:42:38.035382 14312 heartbeater.cc:503] Master 127.1.49.62:44433 was elected leader, sending a full tablet report...
I20260926 13:42:38.038143 14482 tablet_bootstrap.cc:654] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3: Neither blocks nor log segments found. Creating new log.
I20260926 13:42:38.039464 14482 log.cc:824] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:38.041069 14481 tablet_bootstrap.cc:492] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9: No bootstrap required, opened a new log
I20260926 13:42:38.041343 14481 ts_tablet_manager.cc:1397] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.004s
I20260926 13:42:38.047029 14481 raft_consensus.cc:348] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44e85a0cec947ed84501f2eec8f66b9" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42799 } } peers { permanent_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39419 } } peers { permanent_uuid: "3f3e619b47bc4a8ea3290010dc207996" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43961 } }
I20260926 13:42:38.047410 14481 raft_consensus.cc:374] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:42:38.047484 14481 raft_consensus.cc:729] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44e85a0cec947ed84501f2eec8f66b9, State: Initialized, Role: FOLLOWER
I20260926 13:42:38.047640 14482 tablet_bootstrap.cc:492] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3: No bootstrap required, opened a new log
I20260926 13:42:38.048007 14482 ts_tablet_manager.cc:1397] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.002s
I20260926 13:42:38.048096 14481 consensus_queue.cc:260] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44e85a0cec947ed84501f2eec8f66b9" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42799 } } peers { permanent_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39419 } } peers { permanent_uuid: "3f3e619b47bc4a8ea3290010dc207996" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43961 } }
I20260926 13:42:38.049642 14481 ts_tablet_manager.cc:1428] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20260926 13:42:38.050136 14178 heartbeater.cc:503] Master 127.1.49.62:44433 was elected leader, sending a full tablet report...
I20260926 13:42:38.055526 14482 raft_consensus.cc:348] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44e85a0cec947ed84501f2eec8f66b9" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42799 } } peers { permanent_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39419 } } peers { permanent_uuid: "3f3e619b47bc4a8ea3290010dc207996" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43961 } }
I20260926 13:42:38.055917 14482 raft_consensus.cc:374] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:42:38.056137 14482 raft_consensus.cc:729] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8fb6858c3f24d35a9474fbd3cfab3d3, State: Initialized, Role: FOLLOWER
I20260926 13:42:38.056802 14482 consensus_queue.cc:260] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44e85a0cec947ed84501f2eec8f66b9" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42799 } } peers { permanent_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39419 } } peers { permanent_uuid: "3f3e619b47bc4a8ea3290010dc207996" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43961 } }
I20260926 13:42:38.058336 14482 ts_tablet_manager.cc:1428] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260926 13:42:38.058812 14446 heartbeater.cc:503] Master 127.1.49.62:44433 was elected leader, sending a full tablet report...
W20260926 13:42:38.090885 14179 tablet.cc:2369] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:42:38.117957 14313 tablet.cc:2369] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:42:38.134426 14447 tablet.cc:2369] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:42:38.160151 14488 raft_consensus.cc:482] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:42:38.160431 14488 raft_consensus.cc:504] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44e85a0cec947ed84501f2eec8f66b9" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42799 } } peers { permanent_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39419 } } peers { permanent_uuid: "3f3e619b47bc4a8ea3290010dc207996" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43961 } }
I20260926 13:42:38.162003 14488 leader_election.cc:288] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e44e85a0cec947ed84501f2eec8f66b9 (127.1.49.1:42799), 3f3e619b47bc4a8ea3290010dc207996 (127.1.49.2:43961)
I20260926 13:42:38.170670 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:38.163902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:59853 (local address 127.1.49.1:42799)
0926 13:42:38.164026 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:38.164031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:38.164245 (+ 214us) server_negotiation.cc:407] Connection header received
0926 13:42:38.164512 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:38.164517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:38.164578 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:38.164639 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:42:38.165189 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:38.166068 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:38.168510 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:38.169287 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:38.169392 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:38.169482 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:38.170414 (+ 932us) server_negotiation.cc:299] Negotiation successful
0926 13:42:38.170529 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:42:38.172057 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:38.162668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:42799 (local address 127.1.49.3:59853)
0926 13:42:38.164137 (+ 1469us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:38.164157 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:38.164340 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:38.164814 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:38.164818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:38.164840 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:42:38.165018 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:38.165026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:38.166266 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:38.166270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:38.167785 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:38.167832 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:38.169620 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:38.169624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:38.169760 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:38.169790 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 13:42:38.171776 (+ 1986us) client_negotiation.cc:240] Negotiation successful
0926 13:42:38.171884 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"thread_start_us":144,"threads_started":1}
I20260926 13:42:38.174552 14131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb8955c23d14956a5703ecb93e782c0" candidate_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44e85a0cec947ed84501f2eec8f66b9" is_pre_election: true
I20260926 13:42:38.174930 14131 raft_consensus.cc:2454] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8fb6858c3f24d35a9474fbd3cfab3d3 in term 0.
I20260926 13:42:38.175704 14336 leader_election.cc:302] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e44e85a0cec947ed84501f2eec8f66b9, f8fb6858c3f24d35a9474fbd3cfab3d3; no voters:
I20260926 13:42:38.176065 14488 raft_consensus.cc:2781] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:42:38.176141 14488 raft_consensus.cc:482] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:42:38.176179 14488 raft_consensus.cc:3037] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:38.177558 14488 raft_consensus.cc:504] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44e85a0cec947ed84501f2eec8f66b9" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42799 } } peers { permanent_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39419 } } peers { permanent_uuid: "3f3e619b47bc4a8ea3290010dc207996" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43961 } }
I20260926 13:42:38.177958 14488 leader_election.cc:288] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [CANDIDATE]: Term 1 election: Requested vote from peers e44e85a0cec947ed84501f2eec8f66b9 (127.1.49.1:42799), 3f3e619b47bc4a8ea3290010dc207996 (127.1.49.2:43961)
I20260926 13:42:38.178857 14131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb8955c23d14956a5703ecb93e782c0" candidate_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44e85a0cec947ed84501f2eec8f66b9"
I20260926 13:42:38.179031 14131 raft_consensus.cc:3037] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:38.180563 14131 raft_consensus.cc:2454] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8fb6858c3f24d35a9474fbd3cfab3d3 in term 1.
I20260926 13:42:38.181356 14336 leader_election.cc:302] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e44e85a0cec947ed84501f2eec8f66b9, f8fb6858c3f24d35a9474fbd3cfab3d3; no voters:
I20260926 13:42:38.181665 14488 raft_consensus.cc:2781] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:42:38.181888 14488 raft_consensus.cc:686] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [term 1 LEADER]: Becoming Leader. State: Replica: f8fb6858c3f24d35a9474fbd3cfab3d3, State: Running, Role: LEADER
I20260926 13:42:38.182296 14488 consensus_queue.cc:237] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44e85a0cec947ed84501f2eec8f66b9" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42799 } } peers { permanent_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39419 } } peers { permanent_uuid: "3f3e619b47bc4a8ea3290010dc207996" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43961 } }
I20260926 13:42:38.186527 13979 catalog_manager.cc:5495] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8fb6858c3f24d35a9474fbd3cfab3d3 (127.1.49.3). New cstate: current_term: 1 leader_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44e85a0cec947ed84501f2eec8f66b9" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f3e619b47bc4a8ea3290010dc207996" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43961 } health_report { overall_health: UNKNOWN } } }
I20260926 13:42:38.191928 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:38.163687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:38463 (local address 127.1.49.2:43961)
0926 13:42:38.163880 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:38.163886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:38.168391 (+ 4505us) server_negotiation.cc:407] Connection header received
0926 13:42:38.171239 (+ 2848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:38.171242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:38.171323 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:38.171378 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:42:38.172389 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:38.173130 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:38.189728 (+ 16598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:38.190403 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:38.190507 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:38.190574 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:38.191474 (+ 900us) server_negotiation.cc:299] Negotiation successful
0926 13:42:38.191677 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:42:38.192687 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:38.163161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:43961 (local address 127.1.49.3:38463)
0926 13:42:38.168255 (+ 5094us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:38.168279 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:38.169999 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:38.171535 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:38.171538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:38.171559 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:42:38.171671 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:38.171676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:38.173293 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:38.173296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:38.189476 (+ 16180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:38.189496 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:38.191178 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:38.191182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:38.191305 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:38.191331 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 13:42:38.192402 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0926 13:42:38.192509 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4973,"thread_start_us":105,"threads_started":1}
I20260926 13:42:38.194025 14266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb8955c23d14956a5703ecb93e782c0" candidate_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f3e619b47bc4a8ea3290010dc207996" is_pre_election: true
I20260926 13:42:38.194433 14266 raft_consensus.cc:2454] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8fb6858c3f24d35a9474fbd3cfab3d3 in term 0.
I20260926 13:42:38.194823 14265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb8955c23d14956a5703ecb93e782c0" candidate_uuid: "f8fb6858c3f24d35a9474fbd3cfab3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f3e619b47bc4a8ea3290010dc207996"
I20260926 13:42:38.194947 14265 raft_consensus.cc:3037] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:38.196542 14265 raft_consensus.cc:2454] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8fb6858c3f24d35a9474fbd3cfab3d3 in term 1.
I20260926 13:42:38.218376 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:38.212030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.1.49.3:39419)
0926 13:42:38.212181 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:38.212186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:38.212216 (+ 30us) server_negotiation.cc:407] Connection header received
0926 13:42:38.212320 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:38.212323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:38.212400 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:38.212537 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:42:38.212945 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:38.213809 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:38.215360 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:38.215865 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:38.216003 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:38.216687 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:42:38.217374 (+ 687us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:42:38.217533 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:38.218097 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 13:42:38.218177 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:42:38.228641 14131 raft_consensus.cc:1261] T 7eb8955c23d14956a5703ecb93e782c0 P e44e85a0cec947ed84501f2eec8f66b9 [term 1 FOLLOWER]: Refusing update from remote peer f8fb6858c3f24d35a9474fbd3cfab3d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:42:38.229918 14265 raft_consensus.cc:1261] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996 [term 1 FOLLOWER]: Refusing update from remote peer f8fb6858c3f24d35a9474fbd3cfab3d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:42:38.230963 14488 consensus_queue.cc:1035] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f3e619b47bc4a8ea3290010dc207996" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 43961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:42:38.232594 14488 consensus_queue.cc:1035] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e44e85a0cec947ed84501f2eec8f66b9" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:42:38.253268 14494 mvcc.cc:205] Tried to move back new op lower bound from 7333601928089612288 to 7333601927920164864. Current Snapshot: MvccSnapshot[applied={T|T < 7333601928089612288}]
I20260926 13:42:38.254573 14497 mvcc.cc:205] Tried to move back new op lower bound from 7333601928089612288 to 7333601927920164864. Current Snapshot: MvccSnapshot[applied={T|T < 7333601928089612288}]
I20260926 13:42:38.266909 14495 mvcc.cc:205] Tried to move back new op lower bound from 7333601928089612288 to 7333601927920164864. Current Snapshot: MvccSnapshot[applied={T|T < 7333601928089612288}]
I20260926 13:42:38.274991 13978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:42:38.275238 13978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:42:38.280210 13978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26d87cb4daca462683b8834c15530a24: Sending DeleteTablet for 3 replicas of tablet 7eb8955c23d14956a5703ecb93e782c0
I20260926 13:42:38.281350 14106 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eb8955c23d14956a5703ecb93e782c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:42:38 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:58956
I20260926 13:42:38.283329 14377 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eb8955c23d14956a5703ecb93e782c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:42:38 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:39172
I20260926 13:42:38.284824 14246 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eb8955c23d14956a5703ecb93e782c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:42:38 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:46858
I20260926 13:42:38.286594 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 14017
I20260926 13:42:38.292933 14505 tablet_replica.cc:330] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996: stopping tablet replica
I20260926 13:42:38.293558 14505 raft_consensus.cc:2229] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:42:38.293942 14505 raft_consensus.cc:2258] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:42:38.294492 14504 tablet_replica.cc:330] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3: stopping tablet replica
I20260926 13:42:38.294904 14504 raft_consensus.cc:2229] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:42:38.295359 14504 raft_consensus.cc:2258] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:42:38.297530 14504 ts_tablet_manager.cc:1905] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:42:38.302076 14504 ts_tablet_manager.cc:1918] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:42:38.302215 14504 log.cc:1196] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/7eb8955c23d14956a5703ecb93e782c0
I20260926 13:42:38.302805 14504 ts_tablet_manager.cc:1939] T 7eb8955c23d14956a5703ecb93e782c0 P f8fb6858c3f24d35a9474fbd3cfab3d3: Deleting consensus metadata
I20260926 13:42:38.305969 14505 ts_tablet_manager.cc:1905] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 13:42:38.307222 13966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:42799 (error 108)
I20260926 13:42:38.309382 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:38.308968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:42799 (local address 127.0.0.1:58972)
0926 13:42:38.309194 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:38.309277 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260926 13:42:38.309610 13966 catalog_manager.cc:4584] TS e44e85a0cec947ed84501f2eec8f66b9 (127.1.49.1:42799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7eb8955c23d14956a5703ecb93e782c0: Network error: Client connection negotiation failed: client connection to 127.1.49.1:42799: connect: Connection refused (error 111)
I20260926 13:42:38.309998 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 14180
I20260926 13:42:38.310470 14505 ts_tablet_manager.cc:1918] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:42:38.310602 14505 log.cc:1196] T 7eb8955c23d14956a5703ecb93e782c0 P 3f3e619b47bc4a8ea3290010dc207996: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/7eb8955c23d14956a5703ecb93e782c0
I20260926 13:42:38.313588 13963 catalog_manager.cc:4841] TS f8fb6858c3f24d35a9474fbd3cfab3d3 (127.1.49.3:39419): tablet 7eb8955c23d14956a5703ecb93e782c0 (table test-table [id=ee9a3234a96b40649ee959fdaceca4e4]) successfully deleted
I20260926 13:42:38.323889 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 14315
W20260926 13:42:38.323872 14479 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:42:38.323290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:43961 (local address 127.0.0.1:46860)
0926 13:42:38.323428 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:38.323447 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:38.323555 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:38.323668 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.2:43961: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 13:42:38.335059 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 13946
I20260926 13:42:38.345939 1220 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 13:42:38.693558 1220 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 13:42:38.693696 1220 mini_ranger.cc:66] Stopping Ranger...
I20260926 13:42:39.051951 1220 mini_ranger.cc:68] Stopped Ranger
2026-09-26 13:42:39.052 UTC [13013] LOG: received smart shutdown request
2026-09-26 13:42:39.059 UTC [13013] LOG: background worker "logical replication launcher" (PID 13022) exited with exit code 1
2026-09-26 13:42:39.060 UTC [13017] LOG: shutting down
2026-09-26 13:42:39.364 UTC [13013] LOG: database system is shut down
2026-09-26T13:42:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55174 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:42:39 dist-test-slave-1wns krb5kdc[14516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:42:39 dist-test-slave-1wns krb5kdc[14516](info): set up 2 sockets
Sep 26 13:42:39 dist-test-slave-1wns krb5kdc[14516](info): commencing operation
krb5kdc: starting...
W20260926 13:42:40.616196 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:42:40 dist-test-slave-1wns krb5kdc[14516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:42:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:42:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:42:40.764952 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:34967
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:34967,127.1.49.61:33855,127.1.49.60:37031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:42:40.887420 14529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:40.887794 14529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:40.887908 14529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:40.892329 14529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:42:40.892429 14529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:40.892467 14529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:40.892518 14529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:42:40.892555 14529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:42:40.898489 14529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:34967,127.1.49.61:33855,127.1.49.60:37031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:34967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:40.899724 14529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:40.900864 14529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:40.910622 14534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:40.910624 14537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:40.910622 14535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:40.911001 14529 server_base.cc:1034] running on GCE node
I20260926 13:42:40.911900 14529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:40.913321 14529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:40.914589 14529 hybrid_clock.cc:648] HybridClock initialized: now 1790430160914552 us; error 55 us; skew 500 ppm
Sep 26 13:42:40 dist-test-slave-1wns krb5kdc[14516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430160, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:40.918048 14529 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:42:40.919198 14529 webserver.cc:460] Webserver started at http://127.1.49.62:43513/ using document root <none> and password file <none>
I20260926 13:42:40.919926 14529 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:40.920032 14529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:40.920310 14529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:42:40.922904 14529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "b665adecc70f45998046113706b301ca"
format_stamp: "Formatted at 2026-09-26 13:42:40 on dist-test-slave-1wns"
server_key: "825e4db4d97cd48ba3308f99c75a16ae"
server_key_iv: "f8b468a69becd9b68200a2aed23e5c67"
server_key_version: "encryptionkey@0"
I20260926 13:42:40.923609 14529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "b665adecc70f45998046113706b301ca"
format_stamp: "Formatted at 2026-09-26 13:42:40 on dist-test-slave-1wns"
server_key: "825e4db4d97cd48ba3308f99c75a16ae"
server_key_iv: "f8b468a69becd9b68200a2aed23e5c67"
server_key_version: "encryptionkey@0"
I20260926 13:42:40.928205 14529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:42:40.931320 14544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:40.932988 14529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:42:40.933202 14529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "b665adecc70f45998046113706b301ca"
format_stamp: "Formatted at 2026-09-26 13:42:40 on dist-test-slave-1wns"
server_key: "825e4db4d97cd48ba3308f99c75a16ae"
server_key_iv: "f8b468a69becd9b68200a2aed23e5c67"
server_key_version: "encryptionkey@0"
I20260926 13:42:40.933357 14529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:40.958559 14529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:40.962786 14529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:40.963039 14529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:40.971432 14529 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:34967
I20260926 13:42:40.971462 14596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:34967 every 8 connection(s)
I20260926 13:42:40.972699 14529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:42:40.976089 14597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:42:40.980553 14597 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:42:40.983033 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 14529
I20260926 13:42:40.983214 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:42:40.983530 1220 external_mini_cluster.cc:1428] Setting key a874679ef356fea1891aa5b3ed703c84
WARNING: no policy specified for kudu/127.1.49.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.61@KRBTEST.COM" created.
Sep 26 13:42:40 dist-test-slave-1wns krb5kdc[14516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430160, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:41.004681 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:40.984396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.1.49.62:34967)
0926 13:42:40.984796 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:40.984811 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:40.984865 (+ 54us) server_negotiation.cc:407] Connection header received
0926 13:42:40.985446 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:40.985451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:40.985659 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:40.985850 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:40.987160 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:40.988377 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:40.990322 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:40.990836 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:40.990984 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:40.998151 (+ 7167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:40.998164 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:40.998188 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:40.998234 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.000661 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.001466 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.001472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.001473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.001604 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.002113 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.002115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.002117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.002569 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.002719 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.003283 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.003573 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":235,"thread_start_us":106,"threads_started":1}
I20260926 13:42:41.004676 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:40.983338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:34967 (local address 127.0.0.1:60708)
0926 13:42:40.983926 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:40.983988 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:40.985043 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:40.985917 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:40.985926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:40.986515 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:40.986948 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:40.986972 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:40.988524 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:40.988529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:40.990140 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:40.990158 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:40.991054 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:40.991062 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:40.991228 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:40.992079 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:40.992119 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:40.997874 (+ 5755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:41.000875 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.000895 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.000911 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.001315 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.001754 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.001758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.001760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.001947 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.002701 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:41.002710 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:41.002915 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 13:42:41.003149 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 13:42:41.003694 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":110,"thread_start_us":150,"threads_started":1}
I20260926 13:42:41.008582 14597 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:42:41.009534 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.009188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:33855 (local address 127.0.0.1:48350)
0926 13:42:41.009356 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.009448 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 13:42:41.009899 14545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.49.61:33855: connect: Connection refused (error 111)
I20260926 13:42:41.010999 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.010707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:33855 (local address 127.0.0.1:48352)
0926 13:42:41.010835 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.010903 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 13:42:41.011183 14597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.61:33855: Network error: Client connection negotiation failed: client connection to 127.1.49.61:33855: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.61.keytab.
Entry for principal kudu/127.1.49.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.61.keytab.
I20260926 13:42:41.015950 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.61:33855
--webserver_interface=127.1.49.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.61.keytab
--principal=kudu/127.1.49.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:34967,127.1.49.61:33855,127.1.49.60:37031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:42:41.060516 14597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } attempt: 1
I20260926 13:42:41.061570 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.061186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:33855 (local address 127.0.0.1:48358)
0926 13:42:41.061382 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.061466 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 13:42:41.062649 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.062356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:33855 (local address 127.0.0.1:48366)
0926 13:42:41.062487 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.062553 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 13:42:41.062884 14597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.61:33855: Network error: Client connection negotiation failed: client connection to 127.1.49.61:33855: connect: Connection refused (error 111)
I20260926 13:42:41.131197 14597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } attempt: 2
I20260926 13:42:41.132448 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.132055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:33855 (local address 127.0.0.1:48370)
0926 13:42:41.132252 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.132344 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 13:42:41.133318 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.133029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:33855 (local address 127.0.0.1:48372)
0926 13:42:41.133174 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.133241 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 13:42:41.133535 14597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.61:33855: Network error: Client connection negotiation failed: client connection to 127.1.49.61:33855: connect: Connection refused (error 111)
W20260926 13:42:41.149019 14604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:41.149411 14604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:41.149526 14604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:41.154285 14604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:42:41.154397 14604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:41.154435 14604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:41.154476 14604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:42:41.154508 14604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:42:41.160424 14604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:34967,127.1.49.61:33855,127.1.49.60:37031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.61.keytab
--principal=kudu/127.1.49.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.61:33855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.49.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:41.161715 14604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:41.162874 14604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:41.172130 14612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:41.172325 14604 server_base.cc:1034] running on GCE node
W20260926 13:42:41.172142 14610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:41.172130 14609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:41.173321 14604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:41.174703 14604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:41.175907 14604 hybrid_clock.cc:648] HybridClock initialized: now 1790430161175892 us; error 31 us; skew 500 ppm
Sep 26 13:42:41 dist-test-slave-1wns krb5kdc[14516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430161, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:41.179091 14604 init.cc:377] Logged in from keytab as kudu/127.1.49.61@KRBTEST.COM (short username kudu)
I20260926 13:42:41.180182 14604 webserver.cc:460] Webserver started at http://127.1.49.61:32821/ using document root <none> and password file <none>
I20260926 13:42:41.180789 14604 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:41.180883 14604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:41.181078 14604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:42:41.183346 14604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/data/instance:
uuid: "71f182bef28b4a36a441a95a39bcb18b"
format_stamp: "Formatted at 2026-09-26 13:42:41 on dist-test-slave-1wns"
server_key: "a9357a3c342a1e6f6e121a14835a54de"
server_key_iv: "9ba157edfa6dc7def67d787a7f3252c6"
server_key_version: "encryptionkey@0"
I20260926 13:42:41.184010 14604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/wal/instance:
uuid: "71f182bef28b4a36a441a95a39bcb18b"
format_stamp: "Formatted at 2026-09-26 13:42:41 on dist-test-slave-1wns"
server_key: "a9357a3c342a1e6f6e121a14835a54de"
server_key_iv: "9ba157edfa6dc7def67d787a7f3252c6"
server_key_version: "encryptionkey@0"
I20260926 13:42:41.188335 14604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:42:41.191196 14619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:41.193081 14604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:42:41.193239 14604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/wal
uuid: "71f182bef28b4a36a441a95a39bcb18b"
format_stamp: "Formatted at 2026-09-26 13:42:41 on dist-test-slave-1wns"
server_key: "a9357a3c342a1e6f6e121a14835a54de"
server_key_iv: "9ba157edfa6dc7def67d787a7f3252c6"
server_key_version: "encryptionkey@0"
I20260926 13:42:41.193409 14604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:41.212566 14604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:41.216761 14604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:41.216943 14604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:41.224366 14604 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.61:33855
I20260926 13:42:41.224378 14671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.61:33855 every 8 connection(s)
I20260926 13:42:41.224714 14597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } attempt: 3
I20260926 13:42:41.225682 14604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/data/info.pb
I20260926 13:42:41.229394 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 13:42:41 dist-test-slave-1wns krb5kdc[14516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430160, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for kudu/127.1.49.61@KRBTEST.COM
I20260926 13:42:41.234535 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 14604
I20260926 13:42:41.234674 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-1/wal/instance
I20260926 13:42:41.234966 1220 external_mini_cluster.cc:1428] Setting key 831f50161e0034454438303ea9707ef4
I20260926 13:42:41.234783 14672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:42:41.240971 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.225294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:33855 (local address 127.0.0.1:48378)
0926 13:42:41.225508 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.225533 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:41.225666 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:41.228226 (+ 2560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:41.228230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:41.228467 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:41.228623 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.228632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.230487 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.230491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.231422 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.231438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.232385 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.232390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.232521 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.233120 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:41.233137 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:41.235032 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:41.238536 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.238539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.238541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.238883 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.239331 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.239338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.239343 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.239468 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.240425 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:41.240429 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:41.240534 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 13:42:41.240639 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 13:42:41.240819 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 13:42:41.242707 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.226227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48378 (local address 127.1.49.61:33855)
0926 13:42:41.226670 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:41.226690 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:41.226735 (+ 45us) server_negotiation.cc:407] Connection header received
0926 13:42:41.227405 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:41.227421 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:41.227883 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:41.228216 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:41.228959 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.230310 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.231587 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.232232 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.232379 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.235295 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:41.235306 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:41.235327 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:41.235367 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.238332 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.239014 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.239021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.239028 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.239184 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.239582 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.239588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.239591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.240280 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.241238 (+ 958us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.241436 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.241739 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":141,"threads_started":1}
Sep 26 13:42:41 dist-test-slave-1wns krb5kdc[14516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430161, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.61@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:41.245517 14597 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.49.60@KRBTEST.COM; defaulting to no policy
I20260926 13:42:41.246345 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.246059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46080)
0926 13:42:41.246197 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.246260 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 13:42:41.247234 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.246899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46086)
0926 13:42:41.247089 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.247152 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260926 13:42:41.247517 14597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:37031: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.60@KRBTEST.COM" created.
I20260926 13:42:41.250530 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.237318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.1.49.62:34967)
0926 13:42:41.238316 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:41.238321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:41.238333 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:42:41.238372 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:41.238374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:41.238444 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:41.238542 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:41.239366 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.240712 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.242383 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.242767 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.242885 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.245811 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:41.245816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:41.245819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:41.245861 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.248211 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.248901 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.248904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.248906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.248997 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.249308 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.249317 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.249319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.249591 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.249737 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.250205 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.250353 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913}
I20260926 13:42:41.250550 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.237277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:34967 (local address 127.0.0.1:60712)
0926 13:42:41.237774 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.237834 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:41.237974 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:41.238608 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:41.238613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:41.239008 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:41.239226 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.239237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.240853 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.240857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.242205 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.242222 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.243024 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.243028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.243172 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.243854 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:41.243870 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:41.245591 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:41.248388 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.248400 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.248406 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.248781 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.249093 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.249095 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.249097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.249182 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.249761 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:41.249771 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:41.249921 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 13:42:41.250100 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 13:42:41.250356 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":186,"threads_started":1}
I20260926 13:42:41.251983 14672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:42:41 dist-test-slave-1wns krb5kdc[14516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430161, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.61@KRBTEST.COM for kudu/127.1.49.61@KRBTEST.COM
I20260926 13:42:41.262789 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.252697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:33855 (local address 127.0.0.1:48392)
0926 13:42:41.252823 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.252840 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:41.252922 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:41.253187 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:41.253189 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:41.253341 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:41.253445 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.253450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.254674 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.254677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.255282 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.255291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.256122 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.256125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.256426 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.257057 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:41.257070 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:41.258838 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:41.261007 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.261010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.261012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.261233 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.261722 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.261725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.261726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.261820 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.262269 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:41.262272 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:41.262384 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 13:42:41.262488 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 13:42:41.262619 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":4}
I20260926 13:42:41.262825 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.252774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.1.49.61:33855)
0926 13:42:41.252926 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:41.252933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:41.252953 (+ 20us) server_negotiation.cc:407] Connection header received
0926 13:42:41.252999 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:41.253005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:41.253064 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:41.253156 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:41.253559 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.254522 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.255419 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.255967 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.256120 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.259010 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:41.259014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:41.259017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:41.259047 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.260864 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.261362 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.261366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.261367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.261608 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.261913 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.261916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.261917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.262131 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.262275 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.262581 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.262701 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 13:42:41.264310 14672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:42:41.265173 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.264891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46092)
0926 13:42:41.265021 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.265105 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 13:42:41.265497 14622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
I20260926 13:42:41.266500 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.266217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46096)
0926 13:42:41.266353 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.266441 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 13:42:41.266676 14672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:37031: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.60.keytab.
Entry for principal kudu/127.1.49.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.60.keytab.
I20260926 13:42:41.271360 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.60:37031
--webserver_interface=127.1.49.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.60.keytab
--principal=kudu/127.1.49.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:34967,127.1.49.61:33855,127.1.49.60:37031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:42:41.278717 14597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } attempt: 1
I20260926 13:42:41.279887 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.279396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46104)
0926 13:42:41.279611 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.279708 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 13:42:41.280889 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.280544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46114)
0926 13:42:41.280698 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.280776 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 13:42:41.281059 14597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:37031: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
I20260926 13:42:41.315979 14672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } attempt: 1
I20260926 13:42:41.316965 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.316624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46128)
0926 13:42:41.316804 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.316869 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 13:42:41.317858 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.317615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46130)
0926 13:42:41.317728 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.317795 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 13:42:41.318029 14672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:37031: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
I20260926 13:42:41.356269 14597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } attempt: 2
I20260926 13:42:41.357334 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.356941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46140)
0926 13:42:41.357150 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.357232 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 13:42:41.358166 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.357935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46148)
0926 13:42:41.358047 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.358100 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 13:42:41.358398 14597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:37031: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
I20260926 13:42:41.386231 14672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } attempt: 2
I20260926 13:42:41.387223 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.386875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46158)
0926 13:42:41.387060 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.387128 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 13:42:41.388216 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.387963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46160)
0926 13:42:41.388072 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.388133 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 13:42:41.388348 14672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:37031: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
W20260926 13:42:41.404102 14679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:41.404453 14679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:41.404522 14679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:41.408788 14679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:42:41.408881 14679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:41.408918 14679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:41.408973 14679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:42:41.409024 14679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:42:41.414708 14679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:34967,127.1.49.61:33855,127.1.49.60:37031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.60.keytab
--principal=kudu/127.1.49.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.60:37031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.49.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:41.416009 14679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:41.417184 14679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:41.426808 14685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:41.426815 14684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:41.426815 14687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:41.427189 14679 server_base.cc:1034] running on GCE node
I20260926 13:42:41.428010 14679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:41.429340 14679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:41.430562 14679 hybrid_clock.cc:648] HybridClock initialized: now 1790430161430508 us; error 71 us; skew 500 ppm
Sep 26 13:42:41 dist-test-slave-1wns krb5kdc[14516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430161, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:41.433801 14679 init.cc:377] Logged in from keytab as kudu/127.1.49.60@KRBTEST.COM (short username kudu)
I20260926 13:42:41.434875 14679 webserver.cc:460] Webserver started at http://127.1.49.60:33349/ using document root <none> and password file <none>
I20260926 13:42:41.435499 14679 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:41.435607 14679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:41.435905 14679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:42:41.438087 14679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/data/instance:
uuid: "30e571a9b6e44dcf8a0cf153c8580f62"
format_stamp: "Formatted at 2026-09-26 13:42:41 on dist-test-slave-1wns"
server_key: "17a95cb24dbadc9af57f829d5fa5b91a"
server_key_iv: "2d4c798412683fb7da7896a5596f47fb"
server_key_version: "encryptionkey@0"
I20260926 13:42:41.438756 14679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/wal/instance:
uuid: "30e571a9b6e44dcf8a0cf153c8580f62"
format_stamp: "Formatted at 2026-09-26 13:42:41 on dist-test-slave-1wns"
server_key: "17a95cb24dbadc9af57f829d5fa5b91a"
server_key_iv: "2d4c798412683fb7da7896a5596f47fb"
server_key_version: "encryptionkey@0"
I20260926 13:42:41.442970 14679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 13:42:41.446076 14694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:41.447542 14679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:42:41.447723 14679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/wal
uuid: "30e571a9b6e44dcf8a0cf153c8580f62"
format_stamp: "Formatted at 2026-09-26 13:42:41 on dist-test-slave-1wns"
server_key: "17a95cb24dbadc9af57f829d5fa5b91a"
server_key_iv: "2d4c798412683fb7da7896a5596f47fb"
server_key_version: "encryptionkey@0"
I20260926 13:42:41.447922 14679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:41.457643 14597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } attempt: 3
I20260926 13:42:41.458658 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.458253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46162)
0926 13:42:41.458483 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.458558 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 13:42:41.459440 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.459208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46174)
0926 13:42:41.459321 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.459375 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 13:42:41.459585 14597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:37031: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37031: connect: Connection refused (error 111)
I20260926 13:42:41.460638 14679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:41.464368 14679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:41.464586 14679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:41.472409 14679 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.60:37031
I20260926 13:42:41.472431 14746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.60:37031 every 8 connection(s)
I20260926 13:42:41.473506 14679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/data/info.pb
I20260926 13:42:41.476686 14747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:42:41.479061 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 14679
I20260926 13:42:41.479192 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790429983839632-1220-0/minicluster-data/master-2/wal/instance
I20260926 13:42:41.479462 1220 external_mini_cluster.cc:1428] Setting key 3d8376986790f6b0df55a8b7758f9330
I20260926 13:42:41.479524 14672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } attempt: 3
I20260926 13:42:41.481601 14747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:42:41 dist-test-slave-1wns krb5kdc[14516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430161, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.60@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
Sep 26 13:42:41 dist-test-slave-1wns krb5kdc[14516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430161, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.61@KRBTEST.COM for kudu/127.1.49.60@KRBTEST.COM
Sep 26 13:42:41 dist-test-slave-1wns krb5kdc[14516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.60@KRBTEST.COM
I20260926 13:42:41.496802 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.483473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.1.49.62:34967)
0926 13:42:41.483606 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:41.483610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:41.484415 (+ 805us) server_negotiation.cc:407] Connection header received
0926 13:42:41.484574 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:41.484577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:41.484631 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:41.484718 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:41.485441 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.485964 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.487585 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.487937 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.488070 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.491501 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:41.491505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:41.491507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:41.491535 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.493666 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.494722 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.494730 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.494735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.494846 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.495256 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.495259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.495261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.495535 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.495641 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.496410 (+ 769us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.496585 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 13:42:41.496863 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.480253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46186)
0926 13:42:41.480500 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.480519 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:41.480627 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:41.483503 (+ 2876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:41.483506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:41.483771 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:41.483978 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.483993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.486915 (+ 2922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.486920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.487871 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.487886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.488526 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.488530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.488626 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.489497 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:41.489516 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:41.492176 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:41.495060 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.495064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.495070 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.495360 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.495713 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.495716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.495718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.495861 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.496368 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:41.496371 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:41.496472 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 13:42:41.496556 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 13:42:41.496704 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260926 13:42:41.497991 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.481054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.1.49.60:37031)
0926 13:42:41.481562 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:41.481585 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:41.481633 (+ 48us) server_negotiation.cc:407] Connection header received
0926 13:42:41.482554 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:41.482582 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:41.483122 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:41.484488 (+ 1366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:41.484822 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.486662 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.487632 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.488271 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.488421 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.492639 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:41.492643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:41.492645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:41.492670 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.495382 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.496033 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.496036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.496037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.496110 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.496465 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.496469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.496471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.496685 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.497598 (+ 913us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.497667 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.497765 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":455,"server-negotiator.queue_time_us":339,"spinlock_wait_cycles":3200,"thread_start_us":134,"threads_started":1}
I20260926 13:42:41.497995 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.481055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.1.49.60:37031)
0926 13:42:41.481562 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:41.481585 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:41.481633 (+ 48us) server_negotiation.cc:407] Connection header received
0926 13:42:41.482554 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:41.482582 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:41.483122 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:41.484253 (+ 1131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:41.484822 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.486764 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.487998 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.488372 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.488889 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.492420 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:41.492427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:41.492441 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:41.492463 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.494876 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.495497 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.495503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.495504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.495601 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.495967 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.495970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.495972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.496246 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.497202 (+ 956us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.497320 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.497469 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":340,"server-negotiator.queue_time_us":431,"spinlock_wait_cycles":7680,"thread_start_us":158,"threads_started":1}
I20260926 13:42:41.497996 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.483431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:34967 (local address 127.0.0.1:60722)
0926 13:42:41.484283 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.484324 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:41.484475 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:41.484776 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:41.484782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:41.485138 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:41.485302 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.485312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.486074 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.486077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.487438 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.487449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.488123 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.488128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.488416 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.489434 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:41.489459 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:41.491276 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:41.493800 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.493815 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.493832 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.494595 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.494960 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.494965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.494970 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.495154 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.495704 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:41.495710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:41.496110 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 13:42:41.496342 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 13:42:41.496704 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":234,"thread_start_us":101,"threads_started":1}
I20260926 13:42:41.500640 14747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:42:41.503695 14672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b: Bootstrap starting.
Sep 26 13:42:41 dist-test-slave-1wns krb5kdc[14516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430161, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.60@KRBTEST.COM for kudu/127.1.49.61@KRBTEST.COM
Sep 26 13:42:41 dist-test-slave-1wns krb5kdc[14516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:41.510118 14672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b: Neither blocks nor log segments found. Creating new log.
I20260926 13:42:41.511195 14672 log.cc:824] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:41.511866 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.501178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:33855 (local address 127.0.0.1:48404)
0926 13:42:41.501445 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.501460 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:41.501550 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:41.501787 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:41.501790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:41.501994 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:41.502092 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.502098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.502872 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.502876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.503735 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.503747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.504514 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.504518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.504623 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.505207 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:41.505227 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:41.507028 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:41.509957 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.509961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.509963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.510187 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.510571 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.510576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.510578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.510686 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.511245 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:41.511248 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:41.511359 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 13:42:41.511490 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 13:42:41.511669 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260926 13:42:41.511866 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.501439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.1.49.61:33855)
0926 13:42:41.501576 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:41.501583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:41.501598 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:42:41.501639 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:41.501641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:41.501684 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:41.501773 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:41.502206 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.502749 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.503936 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.504259 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.504340 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.507242 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:41.507246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:41.507252 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:41.507305 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.509824 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.510329 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.510333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.510335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.510452 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.510807 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.510813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.510815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.511080 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.511160 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.511554 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.511669 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:42:41.512609 14747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:42:41.512964 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.501189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.1.49.62:34967)
0926 13:42:41.501311 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:41.501316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:41.501594 (+ 278us) server_negotiation.cc:407] Connection header received
0926 13:42:41.501640 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:41.501642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:41.501690 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:41.501774 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:41.502718 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.503278 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.504180 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.504413 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.504518 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.508461 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:41.508466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:41.508470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:41.508535 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.510335 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.510850 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.510853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.510856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.510955 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.511275 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.511279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.511281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.511524 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.511783 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.512644 (+ 861us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.512798 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:42:41.513959 14672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b: No bootstrap required, opened a new log
I20260926 13:42:41.518558 14672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } }
I20260926 13:42:41.518816 14672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:42:41.518882 14672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f182bef28b4a36a441a95a39bcb18b, State: Initialized, Role: FOLLOWER
Sep 26 13:42:41 dist-test-slave-1wns krb5kdc[14516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430161, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.60@KRBTEST.COM for kudu/127.1.49.60@KRBTEST.COM
I20260926 13:42:41.519496 14672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } }
I20260926 13:42:41.520159 14754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } } }
Sep 26 13:42:41 dist-test-slave-1wns krb5kdc[14516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.61@KRBTEST.COM
I20260926 13:42:41.520432 14754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:42:41.521327 14672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:42:41.523658 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.513219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46216)
0926 13:42:41.513441 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.513455 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:41.513532 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:41.514165 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:41.514167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:41.514302 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:41.514364 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.514368 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.515262 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.515268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.516043 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.516053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.516692 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.516695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.516783 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.517773 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:41.517789 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:41.519403 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:41.521243 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.521247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.521249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.521856 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.522326 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.522333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.522337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.522461 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.522970 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:41.522974 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:41.523087 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 13:42:41.523214 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 13:42:41.523391 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260926 13:42:41.524255 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.513459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46216 (local address 127.1.49.60:37031)
0926 13:42:41.513671 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:41.513675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:41.513685 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:42:41.513713 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:41.513716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:41.513753 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:41.514049 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:41.514492 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.515162 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.516178 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.516558 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.516674 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.519537 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:41.519540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:41.519543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:41.519569 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.521081 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.521984 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.521991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.521994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.522092 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.522562 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.522568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.522572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.522832 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.522955 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.523926 (+ 971us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.524074 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
W20260926 13:42:41.527493 14765 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:42:41.527590 14765 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:42:41.527621 14747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62: Bootstrap starting.
I20260926 13:42:41.525964 14604 master_runner.cc:186] Error getting master registration for 127.1.49.62:34967: OK, instance_id { permanent_uuid: "b665adecc70f45998046113706b301ca" instance_seqno: 1790430160969101 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:42:41.529838 14604 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:42:41.530674 14747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62: Neither blocks nor log segments found. Creating new log.
I20260926 13:42:41.531675 14747 log.cc:824] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:41.532650 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.514497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.1.49.61:33855)
0926 13:42:41.514608 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:41.514612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:41.514644 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:42:41.514676 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:41.514679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:41.514720 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:41.514792 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:41.515576 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.516303 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.517257 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.517512 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.517622 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.521214 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:41.521218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:41.521222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:41.521252 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.523013 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.523585 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.523589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.523591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.523694 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.524281 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.524284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.524286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.524497 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.524727 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.532317 (+ 7590us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.532476 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:42:41.534240 14747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62: No bootstrap required, opened a new log
I20260926 13:42:41.536437 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:42:41.538049 14747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } }
I20260926 13:42:41.538285 14747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:42:41.538322 14747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e571a9b6e44dcf8a0cf153c8580f62, State: Initialized, Role: FOLLOWER
I20260926 13:42:41.538826 14747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } }
I20260926 13:42:41.539373 14772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } } }
I20260926 13:42:41.539626 14772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:42:41.540279 14747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:42:41.544934 14679 master_runner.cc:186] Error getting master registration for 127.1.49.62:34967: OK, instance_id { permanent_uuid: "b665adecc70f45998046113706b301ca" instance_seqno: 1790430160969101 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:42:41.549501 14679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:42:41.551398 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.540347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.1.49.62:34967)
0926 13:42:41.540485 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:41.540490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:41.541144 (+ 654us) server_negotiation.cc:407] Connection header received
0926 13:42:41.541380 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:41.541384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:41.541441 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:41.541511 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:41.542590 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.543388 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.544740 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.545048 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.545140 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.547169 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:41.547174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:41.547177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:41.547324 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.549070 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.549678 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.549681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.549683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.549776 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.550096 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.550100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.550102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.550360 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.550486 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.551070 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.551229 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:42:41.553851 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.540340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.1.49.61:33855)
0926 13:42:41.540490 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:41.540495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:41.541953 (+ 1458us) server_negotiation.cc:407] Connection header received
0926 13:42:41.542179 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:41.542186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:41.542273 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:41.542344 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:41.543299 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.544162 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.545439 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.546226 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.546312 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.549107 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:41.549114 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:41.549117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:41.549154 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.551472 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.552134 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.552138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.552141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.552242 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.552601 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.552605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.552607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.552884 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.553112 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.553505 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.553638 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 13:42:41.554490 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.541174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46228 (local address 127.1.49.60:37031)
0926 13:42:41.541336 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:41.541342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:41.543508 (+ 2166us) server_negotiation.cc:407] Connection header received
0926 13:42:41.543681 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:41.543685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:41.543736 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:41.543787 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:41.544844 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.545640 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.546876 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.547290 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.547389 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.549394 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:41.549424 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:41.549429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:41.549464 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.551422 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.552852 (+ 1430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.552856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.552858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.552955 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.553403 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.553409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.553414 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.553646 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.553783 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.554165 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.554310 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 13:42:41.555630 14786 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:42:41.555712 14786 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:42:41.598135 14772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:42:41.598482 14772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } }
I20260926 13:42:41.599910 14772 leader_election.cc:288] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b665adecc70f45998046113706b301ca (127.1.49.62:34967), 71f182bef28b4a36a441a95a39bcb18b (127.1.49.61:33855)
I20260926 13:42:41.600523 14646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f182bef28b4a36a441a95a39bcb18b" is_pre_election: true
I20260926 13:42:41.600919 14646 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30e571a9b6e44dcf8a0cf153c8580f62 in term 0.
I20260926 13:42:41.601410 14695 leader_election.cc:302] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e571a9b6e44dcf8a0cf153c8580f62, 71f182bef28b4a36a441a95a39bcb18b; no voters:
I20260926 13:42:41.601712 14772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:42:41.601820 14772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:42:41.601873 14772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:41.603411 14772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } }
I20260926 13:42:41.603900 14772 leader_election.cc:288] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [CANDIDATE]: Term 1 election: Requested vote from peers b665adecc70f45998046113706b301ca (127.1.49.62:34967), 71f182bef28b4a36a441a95a39bcb18b (127.1.49.61:33855)
I20260926 13:42:41.604636 14646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f182bef28b4a36a441a95a39bcb18b"
I20260926 13:42:41.604797 14646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:41.606305 14646 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30e571a9b6e44dcf8a0cf153c8580f62 in term 1.
I20260926 13:42:41.606640 14695 leader_election.cc:302] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e571a9b6e44dcf8a0cf153c8580f62, 71f182bef28b4a36a441a95a39bcb18b; no voters:
I20260926 13:42:41.606850 14772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:42:41.607122 14772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [term 1 LEADER]: Becoming Leader. State: Replica: 30e571a9b6e44dcf8a0cf153c8580f62, State: Running, Role: LEADER
I20260926 13:42:41.607533 14772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } }
I20260926 13:42:41.609727 14788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30e571a9b6e44dcf8a0cf153c8580f62. Latest consensus state: current_term: 1 leader_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } } }
I20260926 13:42:41.609867 14788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:41.610308 14790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:42:41.611421 14790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:42:41.617470 14646 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [term 1 FOLLOWER]: Refusing update from remote peer 30e571a9b6e44dcf8a0cf153c8580f62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:42:41.618268 14772 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:42:41.623086 14754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30e571a9b6e44dcf8a0cf153c8580f62. Latest consensus state: current_term: 1 leader_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } } }
I20260926 13:42:41.623250 14754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:42:41.623731 14597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } attempt: 4
I20260926 13:42:41.627550 14772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } } }
I20260926 13:42:41.627705 14772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:41.628247 14791 mvcc.cc:205] Tried to move back new op lower bound from 7333601941977436160 to 7333601941947883520. Current Snapshot: MvccSnapshot[applied={T|T < 7333601941977436160}]
Sep 26 13:42:41 dist-test-slave-1wns krb5kdc[14516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430160, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for kudu/127.1.49.60@KRBTEST.COM
I20260926 13:42:41.631824 14754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } } }
I20260926 13:42:41.631966 14754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:42:41.637187 14790 catalog_manager.cc:1347] Generated new cluster ID: ce6ccebfe755477ca1731944cfbcc34d
I20260926 13:42:41.637260 14790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:42:41.643359 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.624422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.1.49.60:37031)
0926 13:42:41.624539 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:41.624544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:41.624742 (+ 198us) server_negotiation.cc:407] Connection header received
0926 13:42:41.624798 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:41.624801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:41.624852 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:41.624927 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:41.625561 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.626294 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.627603 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.627935 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.628038 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.631068 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:41.631072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:41.631074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:41.631105 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:41.633183 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.634090 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.634103 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.634140 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.634342 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:41.634935 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:41.634938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:41.634978 (+ 40us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:41.635480 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:41.635628 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:41.636418 (+ 790us) server_negotiation.cc:299] Negotiation successful
0926 13:42:41.636689 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:42:41.644089 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:41.624334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37031 (local address 127.0.0.1:46232)
0926 13:42:41.624532 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:41.624549 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:41.624648 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:41.625060 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:41.625063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:41.625306 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:41.625423 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.625430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.626497 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.626501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.627401 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:41.627415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:41.628455 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:41.628461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:41.628543 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:41.629219 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:41.629234 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:41.630868 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:41.633382 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.633386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.633389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.633911 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.634570 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:41.634573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:41.634575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:41.634771 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:41.635838 (+ 1067us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:41.635841 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:41.636045 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 13:42:41.636875 (+ 830us) client_negotiation.cc:240] Negotiation successful
0926 13:42:41.643899 (+ 7024us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 13:42:41.646847 14792 mvcc.cc:205] Tried to move back new op lower bound from 7333601941977436160 to 7333601941947883520. Current Snapshot: MvccSnapshot[applied={T|T < 7333601941977436160 or (T in {7333601941977436160})}]
I20260926 13:42:41.650198 14597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca: Bootstrap starting.
I20260926 13:42:41.653916 14597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca: Neither blocks nor log segments found. Creating new log.
I20260926 13:42:41.654901 14597 log.cc:824] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:41.655752 14790 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:42:41.657157 14790 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:42:41.665588 14597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca: No bootstrap required, opened a new log
I20260926 13:42:41.670408 14597 raft_consensus.cc:348] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } }
I20260926 13:42:41.670734 14597 raft_consensus.cc:374] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:42:41.670797 14597 raft_consensus.cc:729] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b665adecc70f45998046113706b301ca, State: Initialized, Role: FOLLOWER
I20260926 13:42:41.672056 14597 consensus_queue.cc:260] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } }
I20260926 13:42:41.672820 14797 sys_catalog.cc:455] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } } }
I20260926 13:42:41.673151 14797 sys_catalog.cc:458] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:42:41.673803 14597 sys_catalog.cc:564] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:42:41.673875 14571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b665adecc70f45998046113706b301ca"
I20260926 13:42:41.674360 14569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:41.674054 14570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b665adecc70f45998046113706b301ca" is_pre_election: true
I20260926 13:42:41.676651 14569 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca [term 1 FOLLOWER]: Refusing update from remote peer 30e571a9b6e44dcf8a0cf153c8580f62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:42:41.683040 14788 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:42:41.686746 14790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62: Generated new TSK 0
I20260926 13:42:41.687661 14790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:42:41.695313 14797 sys_catalog.cc:455] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30e571a9b6e44dcf8a0cf153c8580f62. Latest consensus state: current_term: 1 leader_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } } }
I20260926 13:42:41.695636 14797 sys_catalog.cc:458] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:42:41.695822 14797 sys_catalog.cc:455] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } } }
I20260926 13:42:41.696012 14797 sys_catalog.cc:458] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca [sys.catalog]: This master's current role is: FOLLOWER
W20260926 13:42:41.681389 14808 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:42:41.696188 14808 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:42:41.707545 14810 mvcc.cc:205] Tried to move back new op lower bound from 7333601942104961024 to 7333601941947883520. Current Snapshot: MvccSnapshot[applied={T|T < 7333601941977436160}]
I20260926 13:42:41.713567 14772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b665adecc70f45998046113706b301ca" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 34967 } } peers { permanent_uuid: "71f182bef28b4a36a441a95a39bcb18b" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 33855 } } peers { permanent_uuid: "30e571a9b6e44dcf8a0cf153c8580f62" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37031 } } }
I20260926 13:42:41.713748 14772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e571a9b6e44dcf8a0cf153c8580f62 [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:42.529057 14765 catalog_manager.cc:1259] Loaded cluster ID: ce6ccebfe755477ca1731944cfbcc34d
I20260926 13:42:42.529151 14765 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b: loading cluster ID for follower catalog manager: success
I20260926 13:42:42.530885 14765 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b: acquiring CA information for follower catalog manager: success
I20260926 13:42:42.532302 14765 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 71f182bef28b4a36a441a95a39bcb18b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:42:42.698145 14808 catalog_manager.cc:1259] Loaded cluster ID: ce6ccebfe755477ca1731944cfbcc34d
I20260926 13:42:42.698232 14808 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca: loading cluster ID for follower catalog manager: success
I20260926 13:42:42.700162 14808 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca: acquiring CA information for follower catalog manager: success
I20260926 13:42:42.701851 14808 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b665adecc70f45998046113706b301ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:42:42.777122 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 14529
I20260926 13:42:42.783849 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 14604
I20260926 13:42:42.790439 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 14679
2026-09-26T13:42:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3285 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:42:42 dist-test-slave-1wns krb5kdc[14823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:42:42 dist-test-slave-1wns krb5kdc[14823](info): set up 2 sockets
Sep 26 13:42:42 dist-test-slave-1wns krb5kdc[14823](info): commencing operation
krb5kdc: starting...
W20260926 13:42:43.916270 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:42:44 dist-test-slave-1wns krb5kdc[14823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:42:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:42:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:42:44.064072 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:36021
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:35589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:36021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:42:44.189793 14836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:44.190151 14836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:44.190234 14836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:44.194653 14836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:42:44.194780 14836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:44.194816 14836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:44.194854 14836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:42:44.194905 14836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:42:44.200994 14836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:35589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:36021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:36021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:44.202310 14836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:44.203444 14836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:44.212913 14842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:44.213021 14836 server_base.cc:1034] running on GCE node
W20260926 13:42:44.213178 14844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:44.212916 14841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:44.213958 14836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:44.215191 14836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:44.216395 14836 hybrid_clock.cc:648] HybridClock initialized: now 1790430164216374 us; error 37 us; skew 500 ppm
Sep 26 13:42:44 dist-test-slave-1wns krb5kdc[14823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:44.219473 14836 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:42:44.220496 14836 webserver.cc:460] Webserver started at http://127.1.49.62:33555/ using document root <none> and password file <none>
I20260926 13:42:44.221101 14836 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:44.221165 14836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:44.221354 14836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:42:44.223479 14836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "bd1ec391b6f04b929db3d4e0fb7d8bb2"
format_stamp: "Formatted at 2026-09-26 13:42:44 on dist-test-slave-1wns"
server_key: "1f5e242bb95846127a889f7af7134efd"
server_key_iv: "e3a42b352a09c572793138ae0516d681"
server_key_version: "encryptionkey@0"
I20260926 13:42:44.224102 14836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "bd1ec391b6f04b929db3d4e0fb7d8bb2"
format_stamp: "Formatted at 2026-09-26 13:42:44 on dist-test-slave-1wns"
server_key: "1f5e242bb95846127a889f7af7134efd"
server_key_iv: "e3a42b352a09c572793138ae0516d681"
server_key_version: "encryptionkey@0"
I20260926 13:42:44.228281 14836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 13:42:44.231146 14851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:44.232717 14836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:42:44.232884 14836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "bd1ec391b6f04b929db3d4e0fb7d8bb2"
format_stamp: "Formatted at 2026-09-26 13:42:44 on dist-test-slave-1wns"
server_key: "1f5e242bb95846127a889f7af7134efd"
server_key_iv: "e3a42b352a09c572793138ae0516d681"
server_key_version: "encryptionkey@0"
I20260926 13:42:44.233022 14836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:44.259622 14836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:44.263428 14836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:44.263665 14836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:44.271448 14836 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:36021
I20260926 13:42:44.271481 14903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:36021 every 8 connection(s)
I20260926 13:42:44.272604 14836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:42:44.275707 14904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:42:44.282315 14904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2: Bootstrap starting.
I20260926 13:42:44.282492 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 14836
I20260926 13:42:44.282608 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:42:44.282982 1220 external_mini_cluster.cc:1428] Setting key 35740e0193726c3850a2b550dd3964d7
I20260926 13:42:44.285545 14904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2: Neither blocks nor log segments found. Creating new log.
I20260926 13:42:44.286442 14904 log.cc:824] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:44.288699 14904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2: No bootstrap required, opened a new log
I20260926 13:42:44.293007 14904 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1ec391b6f04b929db3d4e0fb7d8bb2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36021 } }
I20260926 13:42:44.293287 14904 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 26 13:42:44 dist-test-slave-1wns krb5kdc[14823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:44.293351 14904 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1ec391b6f04b929db3d4e0fb7d8bb2, State: Initialized, Role: FOLLOWER
I20260926 13:42:44.293830 14904 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1ec391b6f04b929db3d4e0fb7d8bb2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36021 } }
I20260926 13:42:44.293990 14904 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:42:44.294077 14904 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:42:44.294222 14904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:44.295547 14904 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1ec391b6f04b929db3d4e0fb7d8bb2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36021 } }
I20260926 13:42:44.295980 14904 leader_election.cc:302] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd1ec391b6f04b929db3d4e0fb7d8bb2; no voters:
I20260926 13:42:44.296365 14904 leader_election.cc:288] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:42:44.296505 14909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:42:44.296765 14909 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [term 1 LEADER]: Becoming Leader. State: Replica: bd1ec391b6f04b929db3d4e0fb7d8bb2, State: Running, Role: LEADER
I20260926 13:42:44.297180 14909 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1ec391b6f04b929db3d4e0fb7d8bb2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36021 } }
I20260926 13:42:44.297860 14904 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:42:44.299405 14911 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd1ec391b6f04b929db3d4e0fb7d8bb2. Latest consensus state: current_term: 1 leader_uuid: "bd1ec391b6f04b929db3d4e0fb7d8bb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1ec391b6f04b929db3d4e0fb7d8bb2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36021 } } }
I20260926 13:42:44.299572 14911 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:44.299932 14910 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd1ec391b6f04b929db3d4e0fb7d8bb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1ec391b6f04b929db3d4e0fb7d8bb2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36021 } } }
I20260926 13:42:44.300053 14910 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:44.301697 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:44.284534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.1.49.62:36021)
0926 13:42:44.285032 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:44.285049 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:44.285087 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:42:44.285961 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:44.285983 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:44.286376 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:44.286851 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:44.287746 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:44.289017 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:44.290012 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:44.290817 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:44.291032 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:44.294073 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:44.294082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:44.294104 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:44.294168 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:44.296677 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:44.297356 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:44.297362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:44.297369 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:44.297533 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:44.297900 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:44.297904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:44.297906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:44.298527 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:44.298779 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:44.299946 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0926 13:42:44.300378 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":160,"threads_started":1}
W20260926 13:42:44.303460 14924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:42:44.303557 14924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:42:44.303625 14920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:42:44.304556 14920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:42:44.305166 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:42:44.310977 14920 catalog_manager.cc:1347] Generated new cluster ID: 688130ced36449c4a58af5f3d96a6d87
I20260926 13:42:44.311077 14920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:42:44.323711 14920 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:42:44.325210 14920 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:42:44.336586 14920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd1ec391b6f04b929db3d4e0fb7d8bb2: Generated new TSK 0
I20260926 13:42:44.337720 14920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:42:44.362211 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 14836
2026-09-26T13:42:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1568 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:42:44 dist-test-slave-1wns krb5kdc[14931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:42:44 dist-test-slave-1wns krb5kdc[14931](info): set up 2 sockets
Sep 26 13:42:44 dist-test-slave-1wns krb5kdc[14931](info): commencing operation
krb5kdc: starting...
W20260926 13:42:45.524292 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:42:45 dist-test-slave-1wns krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:42:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:42:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:42:45.664786 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:46133
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:46133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:42:45.796543 14944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:45.796939 14944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:45.797029 14944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:45.801865 14944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:42:45.801975 14944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:45.802013 14944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:45.802058 14944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:42:45.802104 14944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:42:45.809442 14944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:46133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:46133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:45.810869 14944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:45.812160 14944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:45.822604 14950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:45.822641 14949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:45.822695 14952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:45.823457 14944 server_base.cc:1034] running on GCE node
I20260926 13:42:45.824290 14944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:45.825644 14944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:45.826890 14944 hybrid_clock.cc:648] HybridClock initialized: now 1790430165826874 us; error 36 us; skew 500 ppm
Sep 26 13:42:45 dist-test-slave-1wns krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430165, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:45.841197 14944 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:42:45.842672 14944 webserver.cc:460] Webserver started at http://127.1.49.62:36739/ using document root <none> and password file <none>
I20260926 13:42:45.843375 14944 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:45.843497 14944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:45.843762 14944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:42:45.846446 14944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "f75e027e2f01413493dc55271df2c7bc"
format_stamp: "Formatted at 2026-09-26 13:42:45 on dist-test-slave-1wns"
server_key: "7c25c5c58448f56724689db4c7a34fdf"
server_key_iv: "97eac7c0dbb3311b3fc04d3c9c81e65e"
server_key_version: "encryptionkey@0"
I20260926 13:42:45.847199 14944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "f75e027e2f01413493dc55271df2c7bc"
format_stamp: "Formatted at 2026-09-26 13:42:45 on dist-test-slave-1wns"
server_key: "7c25c5c58448f56724689db4c7a34fdf"
server_key_iv: "97eac7c0dbb3311b3fc04d3c9c81e65e"
server_key_version: "encryptionkey@0"
I20260926 13:42:45.851886 14944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260926 13:42:45.855196 14959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:45.856913 14944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:42:45.857128 14944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "f75e027e2f01413493dc55271df2c7bc"
format_stamp: "Formatted at 2026-09-26 13:42:45 on dist-test-slave-1wns"
server_key: "7c25c5c58448f56724689db4c7a34fdf"
server_key_iv: "97eac7c0dbb3311b3fc04d3c9c81e65e"
server_key_version: "encryptionkey@0"
I20260926 13:42:45.857298 14944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:45.886188 14944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:45.890599 14944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:45.890894 14944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:45.899868 14944 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:46133
I20260926 13:42:45.900261 15011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:46133 every 8 connection(s)
I20260926 13:42:45.901110 14944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:42:45.904198 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 14944
I20260926 13:42:45.904359 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:42:45.904574 15012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:42:45.904711 1220 external_mini_cluster.cc:1428] Setting key 560fefefae62df4d0e42b79eed8965f5
I20260926 13:42:45.912425 15012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Bootstrap starting.
I20260926 13:42:45.915110 15012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Neither blocks nor log segments found. Creating new log.
Sep 26 13:42:45 dist-test-slave-1wns krb5kdc[14931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:45.916015 15012 log.cc:824] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:45.918833 15012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: No bootstrap required, opened a new log
I20260926 13:42:45.923019 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:45.906474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.1.49.62:46133)
0926 13:42:45.907046 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:45.907062 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:45.907106 (+ 44us) server_negotiation.cc:407] Connection header received
0926 13:42:45.908452 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:45.908482 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:45.908935 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:45.909375 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:45.910370 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:45.911624 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:45.912788 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:45.913356 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:45.913574 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:45.916645 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:45.916651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:45.916668 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:45.916698 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:45.919019 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:45.919601 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:45.919608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:45.919615 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:45.919780 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:45.920106 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:45.920109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:45.920111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:45.920698 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:45.920941 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:45.921298 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 13:42:45.921735 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":177,"threads_started":1}
I20260926 13:42:45.924670 15012 raft_consensus.cc:348] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } }
I20260926 13:42:45.924985 15012 raft_consensus.cc:374] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:42:45.925027 15012 raft_consensus.cc:729] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75e027e2f01413493dc55271df2c7bc, State: Initialized, Role: FOLLOWER
I20260926 13:42:45.925618 15012 consensus_queue.cc:260] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } }
I20260926 13:42:45.925776 15012 raft_consensus.cc:388] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:42:45.925894 15012 raft_consensus.cc:482] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:42:45.926033 15012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:45.927268 15012 raft_consensus.cc:504] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } }
I20260926 13:42:45.927703 15012 leader_election.cc:302] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f75e027e2f01413493dc55271df2c7bc; no voters:
I20260926 13:42:45.928120 15012 leader_election.cc:288] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:42:45.928309 15017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:42:45.928565 15017 raft_consensus.cc:686] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 1 LEADER]: Becoming Leader. State: Replica: f75e027e2f01413493dc55271df2c7bc, State: Running, Role: LEADER
I20260926 13:42:45.928925 15017 consensus_queue.cc:237] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } }
I20260926 13:42:45.929617 15012 sys_catalog.cc:564] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:42:45.931015 15018 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f75e027e2f01413493dc55271df2c7bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } } }
I20260926 13:42:45.931188 15018 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:45.931034 15019 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75e027e2f01413493dc55271df2c7bc. Latest consensus state: current_term: 1 leader_uuid: "f75e027e2f01413493dc55271df2c7bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } } }
I20260926 13:42:45.931293 15019 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:45.931634 15026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:42:45.935392 15026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:42:45.942516 15026 catalog_manager.cc:1347] Generated new cluster ID: b19507baf68c4ff7a990139f79660f3d
I20260926 13:42:45.942631 15026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:42:45.980460 15026 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:42:45.982036 15026 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:42:45.993129 15026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Generated new TSK 0
I20260926 13:42:45.994182 15026 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:42:46.040285 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:46133
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:42:46.171763 15038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:46.172114 15038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:46.172168 15038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:46.176676 15038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:46.176798 15038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:46.176927 15038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:42:46.182777 15038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:46133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:46.184089 15038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:46.185230 15038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:46.196408 15046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:46.196563 15044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:46.196547 15038 server_base.cc:1034] running on GCE node
W20260926 13:42:46.196439 15043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:46.197113 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:46.197904 15038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:46.199106 15038 hybrid_clock.cc:648] HybridClock initialized: now 1790430166199085 us; error 35 us; skew 500 ppm
Sep 26 13:42:46 dist-test-slave-1wns krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430166, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:46.202960 15038 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:42:46.204226 15038 webserver.cc:460] Webserver started at http://127.1.49.1:46023/ using document root <none> and password file <none>
I20260926 13:42:46.204908 15038 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:46.205013 15038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:46.205286 15038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:42:46.208148 15038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "7a1e24782402468dbfb5343bd5dcc867"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "5cc8876fb6104a75b7116235e10eb378"
server_key_iv: "45647784197308a69b6b5bd7f2b81617"
server_key_version: "encryptionkey@0"
I20260926 13:42:46.208868 15038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "7a1e24782402468dbfb5343bd5dcc867"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "5cc8876fb6104a75b7116235e10eb378"
server_key_iv: "45647784197308a69b6b5bd7f2b81617"
server_key_version: "encryptionkey@0"
I20260926 13:42:46.213461 15038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:42:46.216595 15053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:46.218102 15038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:42:46.218356 15038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "7a1e24782402468dbfb5343bd5dcc867"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "5cc8876fb6104a75b7116235e10eb378"
server_key_iv: "45647784197308a69b6b5bd7f2b81617"
server_key_version: "encryptionkey@0"
I20260926 13:42:46.218525 15038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:46.249552 15038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:46.253207 15038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:46.253448 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:46.254099 15038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:46.255008 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:42:46.255079 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:46.255149 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:42:46.255188 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:46.265885 15038 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:42311
I20260926 13:42:46.265903 15166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:42311 every 8 connection(s)
I20260926 13:42:46.266930 15038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:42:46.268525 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 15038
I20260926 13:42:46.268704 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:42:46.269040 1220 external_mini_cluster.cc:1428] Setting key 76e2ad459c3a605f9d3b481fcb249952
Sep 26 13:42:46 dist-test-slave-1wns krb5kdc[14931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430166, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
I20260926 13:42:46.285697 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.269242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:53693 (local address 127.1.49.62:46133)
0926 13:42:46.269425 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:46.269431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:46.270926 (+ 1495us) server_negotiation.cc:407] Connection header received
0926 13:42:46.272038 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:46.272042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:46.272105 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:46.272207 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:46.273799 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.274409 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.276867 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.277129 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.277239 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.280824 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:46.280828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:46.280830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:46.280860 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:46.283053 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:46.283940 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:46.283945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:46.283946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:46.284037 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:46.284428 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:46.284432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:46.284434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:46.284649 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:46.284761 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:46.285339 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 13:42:46.285501 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 13:42:46.286631 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.269857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:53693)
0926 13:42:46.270747 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:46.270816 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:46.271738 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:46.272452 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:46.272460 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:46.272903 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:46.273561 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.273590 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.274577 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.274585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.276619 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.276641 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.277340 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.277347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.277626 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.278454 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:46.278484 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:46.280588 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:46.283322 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:46.283329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:46.283348 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:46.283742 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:46.284145 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:46.284148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:46.284150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:46.284320 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:46.284766 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:46.284776 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:46.285068 (+ 292us) client_negotiation.cc:769] Sending connection context
0926 13:42:46.285283 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 13:42:46.285661 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":141,"threads_started":1}
I20260926 13:42:46.288267 15167 heartbeater.cc:347] Connected to a master server at 127.1.49.62:46133
I20260926 13:42:46.288564 15167 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:46.289150 15167 heartbeater.cc:511] Master 127.1.49.62:46133 requested a full tablet report, sending...
I20260926 13:42:46.291252 14972 ts_manager.cc:194] Registered new tserver with Master: 7a1e24782402468dbfb5343bd5dcc867 (127.1.49.1:42311)
I20260926 13:42:46.292937 14972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:53693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:42:46.305066 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:46133
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:42:46.427017 15172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:46.427347 15172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:46.427394 15172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:46.432165 15172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:46.432241 15172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:46.432327 15172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:42:46.438086 15172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:46133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:46.439503 15172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:46.440840 15172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:46.451753 15177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:46.451910 15178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:46.451937 15172 server_base.cc:1034] running on GCE node
W20260926 13:42:46.451740 15180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:46.452666 15172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:46.453423 15172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:46.454627 15172 hybrid_clock.cc:648] HybridClock initialized: now 1790430166454604 us; error 34 us; skew 500 ppm
Sep 26 13:42:46 dist-test-slave-1wns krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430166, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:46.458074 15172 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:42:46.459244 15172 webserver.cc:460] Webserver started at http://127.1.49.2:41621/ using document root <none> and password file <none>
I20260926 13:42:46.460019 15172 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:46.460134 15172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:46.460420 15172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:42:46.463032 15172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "067fdeda67394d56be906073b737d5b3"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "a8a606ad51541c83926ece6ad98955c6"
server_key_iv: "6dee53dd1b72ee196245bb0f1f290e88"
server_key_version: "encryptionkey@0"
I20260926 13:42:46.463678 15172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "067fdeda67394d56be906073b737d5b3"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "a8a606ad51541c83926ece6ad98955c6"
server_key_iv: "6dee53dd1b72ee196245bb0f1f290e88"
server_key_version: "encryptionkey@0"
I20260926 13:42:46.468357 15172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 13:42:46.471284 15187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:46.472993 15172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:42:46.473152 15172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "067fdeda67394d56be906073b737d5b3"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "a8a606ad51541c83926ece6ad98955c6"
server_key_iv: "6dee53dd1b72ee196245bb0f1f290e88"
server_key_version: "encryptionkey@0"
I20260926 13:42:46.473294 15172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:46.486836 15172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:46.490768 15172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:46.491022 15172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:46.491727 15172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:46.492798 15172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:42:46.492867 15172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:46.492934 15172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:42:46.492977 15172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:46.503329 15172 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:36147
I20260926 13:42:46.503340 15300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:36147 every 8 connection(s)
I20260926 13:42:46.504470 15172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:42:46.513353 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 15172
I20260926 13:42:46.513481 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:42:46.513794 1220 external_mini_cluster.cc:1428] Setting key 828c2c877b7e36a9b844e440f3a37fec
Sep 26 13:42:46 dist-test-slave-1wns krb5kdc[14931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430166, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:46.521970 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.506685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:45667 (local address 127.1.49.62:46133)
0926 13:42:46.506879 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:46.506885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:46.508032 (+ 1147us) server_negotiation.cc:407] Connection header received
0926 13:42:46.509504 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:46.509507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:46.509562 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:46.509651 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:46.511141 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.511774 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.513305 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.513734 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.513866 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.517105 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:46.517114 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:46.517121 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:46.517162 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:46.519046 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:46.519791 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:46.519794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:46.519795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:46.519957 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:46.520462 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:46.520466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:46.520467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:46.520712 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:46.520808 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:46.521665 (+ 857us) server_negotiation.cc:299] Negotiation successful
0926 13:42:46.521833 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 13:42:46.523244 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.507155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:45667)
0926 13:42:46.507850 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:46.507912 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:46.509266 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:46.509848 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:46.509856 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:46.510392 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:46.510960 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.510976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.511971 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.511974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.513136 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.513148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.513969 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.513980 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.514279 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.515042 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:46.515061 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:46.516904 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:46.519207 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:46.519217 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:46.519239 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:46.519677 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:46.520072 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:46.520078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:46.520081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:46.520316 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:46.520839 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:46.520857 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:46.521283 (+ 426us) client_negotiation.cc:769] Sending connection context
0926 13:42:46.521556 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 13:42:46.522008 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":133,"threads_started":1}
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
I20260926 13:42:46.524874 15301 heartbeater.cc:347] Connected to a master server at 127.1.49.62:46133
I20260926 13:42:46.525281 15301 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:46.525940 15301 heartbeater.cc:511] Master 127.1.49.62:46133 requested a full tablet report, sending...
I20260926 13:42:46.527240 14972 ts_manager.cc:194] Registered new tserver with Master: 067fdeda67394d56be906073b737d5b3 (127.1.49.2:36147)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
I20260926 13:42:46.528162 14972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:45667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:42:46.548704 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:46133
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:42:46.674196 15306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:46.674670 15306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:46.674803 15306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:46.679382 15306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:46.679482 15306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:46.679580 15306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:42:46.685460 15306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:46133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:46.686728 15306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:46.687938 15306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:46.698832 15314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:46.698812 15312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:46.698810 15311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:46.699275 15306 server_base.cc:1034] running on GCE node
I20260926 13:42:46.699745 15306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:46.700531 15306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:46.701777 15306 hybrid_clock.cc:648] HybridClock initialized: now 1790430166701748 us; error 42 us; skew 500 ppm
Sep 26 13:42:46 dist-test-slave-1wns krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430166, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:46.705070 15306 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:42:46.706199 15306 webserver.cc:460] Webserver started at http://127.1.49.3:36049/ using document root <none> and password file <none>
I20260926 13:42:46.706792 15306 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:46.706863 15306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:46.707041 15306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:42:46.709345 15306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "174e702370d844e2b08c810391415346"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "4ef3859951d4517c0aaaabd316213020"
server_key_iv: "515d9c3fb6341df2ea6ac8f4ca04f264"
server_key_version: "encryptionkey@0"
I20260926 13:42:46.710011 15306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "174e702370d844e2b08c810391415346"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "4ef3859951d4517c0aaaabd316213020"
server_key_iv: "515d9c3fb6341df2ea6ac8f4ca04f264"
server_key_version: "encryptionkey@0"
I20260926 13:42:46.714524 15306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:42:46.717528 15321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:46.719357 15306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:42:46.719491 15306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "174e702370d844e2b08c810391415346"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "4ef3859951d4517c0aaaabd316213020"
server_key_iv: "515d9c3fb6341df2ea6ac8f4ca04f264"
server_key_version: "encryptionkey@0"
I20260926 13:42:46.719609 15306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:46.738979 15306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:46.743109 15306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:46.743359 15306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:46.744148 15306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:46.745133 15306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:42:46.745187 15306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:46.745234 15306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:42:46.745251 15306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:46.756273 15306 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:35239
I20260926 13:42:46.756376 15434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:35239 every 8 connection(s)
I20260926 13:42:46.757378 15306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:42:46.766788 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 15306
I20260926 13:42:46.766971 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:42:46.767315 1220 external_mini_cluster.cc:1428] Setting key 64d9afb37bfe7b56208081f93c0b1a0a
Sep 26 13:42:46 dist-test-slave-1wns krb5kdc[14931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430166, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:46.774916 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.759592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:44203 (local address 127.1.49.62:46133)
0926 13:42:46.759776 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:46.759781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:46.761075 (+ 1294us) server_negotiation.cc:407] Connection header received
0926 13:42:46.762147 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:46.762152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:46.762250 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:46.762362 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:46.764163 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.764814 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.766333 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.766704 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.766899 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.770792 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:46.770795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:46.770797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:46.770825 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:46.772450 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:46.773042 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:46.773045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:46.773047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:46.773136 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:46.773591 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:46.773594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:46.773596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:46.773837 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:46.773924 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:46.774625 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 13:42:46.774775 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 13:42:46.776067 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.760113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:44203)
0926 13:42:46.760883 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:46.760937 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:46.761901 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:46.762624 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:46.762632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:46.763081 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:46.763949 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.763971 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.764984 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.764989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.766121 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.766133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.766986 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.766992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.767278 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.768134 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:46.768195 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:46.770582 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:46.772585 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:46.772592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:46.772607 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:46.772904 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:46.773261 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:46.773264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:46.773266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:46.773453 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:46.773954 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:46.773960 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:46.774348 (+ 388us) client_negotiation.cc:769] Sending connection context
0926 13:42:46.774550 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 13:42:46.774889 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":127,"threads_started":1}
I20260926 13:42:46.777547 15435 heartbeater.cc:347] Connected to a master server at 127.1.49.62:46133
I20260926 13:42:46.777856 15435 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:46.778682 15435 heartbeater.cc:511] Master 127.1.49.62:46133 requested a full tablet report, sending...
I20260926 13:42:46.780099 14972 ts_manager.cc:194] Registered new tserver with Master: 174e702370d844e2b08c810391415346 (127.1.49.3:35239)
I20260926 13:42:46.780786 14972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:44203
I20260926 13:42:46.782400 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:42:46.793095 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.784610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.1.49.62:46133)
0926 13:42:46.784756 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:46.784760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:46.785010 (+ 250us) server_negotiation.cc:407] Connection header received
0926 13:42:46.785114 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:46.785118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:46.785169 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:46.785244 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:46.786047 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.786627 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.787643 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.787958 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.788082 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.789621 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:46.789626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:46.789628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:46.789666 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:46.791123 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:46.791688 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:46.791692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:46.791694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:46.791788 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:46.792121 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:46.792125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:46.792127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:46.792386 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:46.792528 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:46.792792 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 13:42:46.792920 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:42:46.797287 14976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:42:46.800611 14976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:42:46.816958 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.810463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.1.49.3:35239)
0926 13:42:46.811017 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:46.811035 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:46.811105 (+ 70us) server_negotiation.cc:407] Connection header received
0926 13:42:46.811202 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:46.811208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:46.811466 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:46.811676 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:42:46.812184 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.813596 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.815250 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.815910 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.816017 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.816189 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:46.816643 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 13:42:46.816788 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":206,"threads_started":1}
I20260926 13:42:46.817492 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.811056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:42311 (local address 127.0.0.1:46640)
0926 13:42:46.811569 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:46.811586 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:46.811678 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:46.812329 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:46.812336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:46.812357 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:42:46.812472 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.812479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.813940 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.813943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.815711 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.815724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.817056 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.817059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.817183 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.817210 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 13:42:46.817280 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 13:42:46.817357 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":62,"thread_start_us":94,"threads_started":1}
I20260926 13:42:46.817693 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.810294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:35239 (local address 127.0.0.1:49402)
0926 13:42:46.810793 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:46.810848 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:46.810987 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:46.811844 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:46.811850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:46.811874 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:42:46.812020 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.812031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.813786 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.813792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.815100 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.815120 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.816256 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.816260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.816382 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.816422 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 13:42:46.816550 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 13:42:46.816714 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":31,"thread_start_us":115,"threads_started":1}
I20260926 13:42:46.818005 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.811357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46640 (local address 127.1.49.1:42311)
0926 13:42:46.811782 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:46.811789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:46.811833 (+ 44us) server_negotiation.cc:407] Connection header received
0926 13:42:46.811914 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:46.811920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:46.812132 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:46.812295 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:42:46.812609 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.813795 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.815911 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.816729 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.816846 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.816987 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:46.817332 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 13:42:46.817461 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":111,"threads_started":1}
I20260926 13:42:46.819132 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.811830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.1.49.2:36147)
0926 13:42:46.812250 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:46.812263 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:46.812284 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:42:46.812356 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:46.812361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:46.812531 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:46.812705 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:42:46.812927 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.814040 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.815978 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.816701 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.816833 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.816991 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:46.818829 (+ 1838us) server_negotiation.cc:299] Negotiation successful
0926 13:42:46.818968 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":107,"threads_started":1}
I20260926 13:42:46.819468 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.811394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:36147 (local address 127.0.0.1:57422)
0926 13:42:46.811998 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:46.812011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:46.812094 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:46.812674 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:46.812677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:46.812694 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:42:46.812802 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.812807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.814209 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.814212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.815758 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.815766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.817367 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.817371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.817482 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.817506 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 13:42:46.817940 (+ 434us) client_negotiation.cc:240] Negotiation successful
0926 13:42:46.818032 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":230,"thread_start_us":86,"threads_started":1}
I20260926 13:42:46.821177 15101 tablet_service.cc:1467] Processing CreateTablet for tablet 52485d4816cc4ca89f128b89a6af2e11 (DEFAULT_TABLE table=test-table [id=fba36d0d3534452abd4eb5fc55215f19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:42:46.821079 15369 tablet_service.cc:1467] Processing CreateTablet for tablet 52485d4816cc4ca89f128b89a6af2e11 (DEFAULT_TABLE table=test-table [id=fba36d0d3534452abd4eb5fc55215f19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:42:46.822310 15101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52485d4816cc4ca89f128b89a6af2e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:42:46.822331 15369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52485d4816cc4ca89f128b89a6af2e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:42:46.823369 15235 tablet_service.cc:1467] Processing CreateTablet for tablet 52485d4816cc4ca89f128b89a6af2e11 (DEFAULT_TABLE table=test-table [id=fba36d0d3534452abd4eb5fc55215f19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:42:46.826074 15235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52485d4816cc4ca89f128b89a6af2e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:42:46.831246 15453 tablet_bootstrap.cc:492] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346: Bootstrap starting.
I20260926 13:42:46.831575 15452 tablet_bootstrap.cc:492] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: Bootstrap starting.
I20260926 13:42:46.834437 15452 tablet_bootstrap.cc:654] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: Neither blocks nor log segments found. Creating new log.
I20260926 13:42:46.834942 15453 tablet_bootstrap.cc:654] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346: Neither blocks nor log segments found. Creating new log.
I20260926 13:42:46.835628 15452 log.cc:824] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:46.835929 15453 log.cc:824] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:46.836406 15454 tablet_bootstrap.cc:492] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3: Bootstrap starting.
I20260926 13:42:46.838264 15452 tablet_bootstrap.cc:492] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: No bootstrap required, opened a new log
I20260926 13:42:46.838351 15453 tablet_bootstrap.cc:492] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346: No bootstrap required, opened a new log
I20260926 13:42:46.838503 15452 ts_tablet_manager.cc:1397] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 13:42:46.838608 15453 ts_tablet_manager.cc:1397] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 13:42:46.840236 15454 tablet_bootstrap.cc:654] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3: Neither blocks nor log segments found. Creating new log.
I20260926 13:42:46.841499 15454 log.cc:824] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:46.842562 15453 raft_consensus.cc:348] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } } peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } }
I20260926 13:42:46.843357 15453 raft_consensus.cc:374] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:42:46.843442 15453 raft_consensus.cc:729] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174e702370d844e2b08c810391415346, State: Initialized, Role: FOLLOWER
I20260926 13:42:46.843842 15452 raft_consensus.cc:348] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } } peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } }
I20260926 13:42:46.844156 15452 raft_consensus.cc:374] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:42:46.844213 15452 raft_consensus.cc:729] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1e24782402468dbfb5343bd5dcc867, State: Initialized, Role: FOLLOWER
I20260926 13:42:46.844645 15453 consensus_queue.cc:260] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } } peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } }
I20260926 13:42:46.844672 15452 consensus_queue.cc:260] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } } peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } }
I20260926 13:42:46.845191 15454 tablet_bootstrap.cc:492] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3: No bootstrap required, opened a new log
I20260926 13:42:46.845479 15454 ts_tablet_manager.cc:1397] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 13:42:46.845715 15167 heartbeater.cc:503] Master 127.1.49.62:46133 was elected leader, sending a full tablet report...
I20260926 13:42:46.845963 15452 ts_tablet_manager.cc:1428] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 13:42:46.846128 15453 ts_tablet_manager.cc:1428] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 13:42:46.846961 15435 heartbeater.cc:503] Master 127.1.49.62:46133 was elected leader, sending a full tablet report...
I20260926 13:42:46.849886 15454 raft_consensus.cc:348] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } } peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } }
I20260926 13:42:46.850266 15454 raft_consensus.cc:374] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:42:46.850335 15454 raft_consensus.cc:729] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067fdeda67394d56be906073b737d5b3, State: Initialized, Role: FOLLOWER
I20260926 13:42:46.850864 15454 consensus_queue.cc:260] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } } peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } }
I20260926 13:42:46.851601 15301 heartbeater.cc:503] Master 127.1.49.62:46133 was elected leader, sending a full tablet report...
I20260926 13:42:46.852037 15454 ts_tablet_manager.cc:1428] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:42:46.943461 15460 raft_consensus.cc:482] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:42:46.943684 15460 raft_consensus.cc:504] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } } peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } }
I20260926 13:42:46.944911 15460 leader_election.cc:288] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 174e702370d844e2b08c810391415346 (127.1.49.3:35239), 7a1e24782402468dbfb5343bd5dcc867 (127.1.49.1:42311)
I20260926 13:42:46.949906 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.945441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:54941 (local address 127.1.49.3:35239)
0926 13:42:46.945588 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:46.945593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:46.945611 (+ 18us) server_negotiation.cc:407] Connection header received
0926 13:42:46.945673 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:46.945720 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:46.945791 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:46.945878 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:42:46.946243 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.946949 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.948615 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.949297 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.949381 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.949440 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:46.949726 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 13:42:46.949793 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 13:42:46.949940 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.945282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:35239 (local address 127.1.49.2:54941)
0926 13:42:46.945433 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:46.945448 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:46.945574 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:46.945922 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:46.945925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:46.945950 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:42:46.946097 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.946103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.947095 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.947100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.948452 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.948465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.949435 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.949439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.949575 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.949610 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 13:42:46.949702 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 13:42:46.949788 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 13:42:46.950747 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.945826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:43701 (local address 127.1.49.1:42311)
0926 13:42:46.945967 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:46.945973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:46.946298 (+ 325us) server_negotiation.cc:407] Connection header received
0926 13:42:46.946394 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:46.946396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:46.946444 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:46.946507 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:42:46.946857 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.947861 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.949275 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.949965 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.950201 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.950283 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:46.950533 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 13:42:46.950614 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 13:42:46.950774 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.945689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:42311 (local address 127.1.49.2:43701)
0926 13:42:46.946194 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:46.946210 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:46.946311 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:46.946539 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:46.946542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:46.946559 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:42:46.946672 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.946686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.947999 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.948002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.949110 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:46.949122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.950292 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.950298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:46.950405 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.950422 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 13:42:46.950513 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 13:42:46.950584 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":158,"threads_started":1}
I20260926 13:42:46.950731 15389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52485d4816cc4ca89f128b89a6af2e11" candidate_uuid: "067fdeda67394d56be906073b737d5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174e702370d844e2b08c810391415346" is_pre_election: true
I20260926 13:42:46.951129 15389 raft_consensus.cc:2454] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 067fdeda67394d56be906073b737d5b3 in term 0.
I20260926 13:42:46.951711 15188 leader_election.cc:302] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067fdeda67394d56be906073b737d5b3, 174e702370d844e2b08c810391415346; no voters:
I20260926 13:42:46.951704 15121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52485d4816cc4ca89f128b89a6af2e11" candidate_uuid: "067fdeda67394d56be906073b737d5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a1e24782402468dbfb5343bd5dcc867" is_pre_election: true
I20260926 13:42:46.952077 15121 raft_consensus.cc:2454] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 067fdeda67394d56be906073b737d5b3 in term 0.
I20260926 13:42:46.952054 15460 raft_consensus.cc:2781] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:42:46.952175 15460 raft_consensus.cc:482] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:42:46.952226 15460 raft_consensus.cc:3037] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:46.953470 15460 raft_consensus.cc:504] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } } peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } }
I20260926 13:42:46.953899 15460 leader_election.cc:288] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [CANDIDATE]: Term 1 election: Requested vote from peers 174e702370d844e2b08c810391415346 (127.1.49.3:35239), 7a1e24782402468dbfb5343bd5dcc867 (127.1.49.1:42311)
I20260926 13:42:46.954424 15389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52485d4816cc4ca89f128b89a6af2e11" candidate_uuid: "067fdeda67394d56be906073b737d5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174e702370d844e2b08c810391415346"
I20260926 13:42:46.954437 15121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52485d4816cc4ca89f128b89a6af2e11" candidate_uuid: "067fdeda67394d56be906073b737d5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a1e24782402468dbfb5343bd5dcc867"
I20260926 13:42:46.954607 15121 raft_consensus.cc:3037] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:46.954587 15389 raft_consensus.cc:3037] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:46.955891 15121 raft_consensus.cc:2454] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 067fdeda67394d56be906073b737d5b3 in term 1.
I20260926 13:42:46.956195 15389 raft_consensus.cc:2454] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 067fdeda67394d56be906073b737d5b3 in term 1.
I20260926 13:42:46.956233 15188 leader_election.cc:302] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067fdeda67394d56be906073b737d5b3, 7a1e24782402468dbfb5343bd5dcc867; no voters:
I20260926 13:42:46.956436 15460 raft_consensus.cc:2781] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:42:46.956691 15460 raft_consensus.cc:686] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [term 1 LEADER]: Becoming Leader. State: Replica: 067fdeda67394d56be906073b737d5b3, State: Running, Role: LEADER
I20260926 13:42:46.957072 15460 consensus_queue.cc:237] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } } peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } }
I20260926 13:42:46.960686 14972 catalog_manager.cc:5495] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 067fdeda67394d56be906073b737d5b3 (127.1.49.2). New cstate: current_term: 1 leader_uuid: "067fdeda67394d56be906073b737d5b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } health_report { overall_health: HEALTHY } } }
I20260926 13:42:46.976938 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:46.972874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.1.49.2:36147)
0926 13:42:46.973008 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:46.973012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:46.973028 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:42:46.973116 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:46.973118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:46.973194 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:46.973307 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:42:46.973528 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.974130 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.975040 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:46.975299 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:46.975410 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:46.975766 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:42:46.976303 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:42:46.976409 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:46.976612 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 13:42:46.976813 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:42:46.984205 15121 raft_consensus.cc:1261] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Refusing update from remote peer 067fdeda67394d56be906073b737d5b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:42:46.984205 15389 raft_consensus.cc:1261] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346 [term 1 FOLLOWER]: Refusing update from remote peer 067fdeda67394d56be906073b737d5b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:42:46.985124 15460 consensus_queue.cc:1035] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:42:46.985379 15462 consensus_queue.cc:1035] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:42:47.005276 14972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:42:47.005501 14972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:42:47.008518 14972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Sending DeleteTablet for 3 replicas of tablet 52485d4816cc4ca89f128b89a6af2e11
I20260926 13:42:47.009644 15369 tablet_service.cc:1514] Processing DeleteTablet for tablet 52485d4816cc4ca89f128b89a6af2e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:42:47 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:49402
I20260926 13:42:47.009651 15235 tablet_service.cc:1514] Processing DeleteTablet for tablet 52485d4816cc4ca89f128b89a6af2e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:42:47 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:57422
I20260926 13:42:47.009660 15101 tablet_service.cc:1514] Processing DeleteTablet for tablet 52485d4816cc4ca89f128b89a6af2e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:42:47 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:46640
I20260926 13:42:47.010218 15475 tablet_replica.cc:330] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346: stopping tablet replica
I20260926 13:42:47.010257 15476 tablet_replica.cc:330] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3: stopping tablet replica
I20260926 13:42:47.010453 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 15038
I20260926 13:42:47.010628 15476 raft_consensus.cc:2229] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:42:47.010752 15475 raft_consensus.cc:2229] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:42:47.011044 15476 raft_consensus.cc:2258] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:42:47.011210 15475 raft_consensus.cc:2258] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:42:47.013141 15475 ts_tablet_manager.cc:1905] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:42:47.013828 15476 ts_tablet_manager.cc:1905] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:42:47.016698 15476 ts_tablet_manager.cc:1918] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:42:47.016824 15476 log.cc:1196] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/52485d4816cc4ca89f128b89a6af2e11
I20260926 13:42:47.017146 15476 ts_tablet_manager.cc:1939] T 52485d4816cc4ca89f128b89a6af2e11 P 067fdeda67394d56be906073b737d5b3: Deleting consensus metadata
I20260926 13:42:47.017167 15475 ts_tablet_manager.cc:1918] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:42:47.017409 15475 log.cc:1196] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/52485d4816cc4ca89f128b89a6af2e11
I20260926 13:42:47.017777 15475 ts_tablet_manager.cc:1939] T 52485d4816cc4ca89f128b89a6af2e11 P 174e702370d844e2b08c810391415346: Deleting consensus metadata
I20260926 13:42:47.019356 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 15172
I20260926 13:42:47.019392 14960 catalog_manager.cc:4841] TS 174e702370d844e2b08c810391415346 (127.1.49.3:35239): tablet 52485d4816cc4ca89f128b89a6af2e11 (table test-table [id=fba36d0d3534452abd4eb5fc55215f19]) successfully deleted
W20260926 13:42:47.019738 14960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:42311 (error 108)
I20260926 13:42:47.020500 14960 catalog_manager.cc:4841] TS 067fdeda67394d56be906073b737d5b3 (127.1.49.2:36147): tablet 52485d4816cc4ca89f128b89a6af2e11 (table test-table [id=fba36d0d3534452abd4eb5fc55215f19]) successfully deleted
I20260926 13:42:47.022466 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.022124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:42311 (local address 127.0.0.1:46648)
0926 13:42:47.022270 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:47.022374 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:42311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 13:42:47.022677 14960 catalog_manager.cc:4584] TS 7a1e24782402468dbfb5343bd5dcc867 (127.1.49.1:42311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 52485d4816cc4ca89f128b89a6af2e11: Network error: Client connection negotiation failed: client connection to 127.1.49.1:42311: connect: Connection refused (error 111)
I20260926 13:42:47.025743 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 15306
I20260926 13:42:47.031949 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 14944
I20260926 13:42:47.039076 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:46133
--webserver_interface=127.1.49.62
--webserver_port=36739
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:46133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:42:47.161660 15479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:47.161978 15479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:47.162024 15479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:47.166330 15479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:42:47.166411 15479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:47.166432 15479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:47.166450 15479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:42:47.166466 15479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:42:47.172443 15479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:46133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:46133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=36739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:47.173619 15479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:47.174736 15479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:47.192257 15486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:47.192265 15485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:47.192759 15488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:47.193146 15479 server_base.cc:1034] running on GCE node
I20260926 13:42:47.193876 15479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:47.195318 15479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:47.196519 15479 hybrid_clock.cc:648] HybridClock initialized: now 1790430167196505 us; error 30 us; skew 500 ppm
Sep 26 13:42:47 dist-test-slave-1wns krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:47.199659 15479 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:42:47.200770 15479 webserver.cc:460] Webserver started at http://127.1.49.62:36739/ using document root <none> and password file <none>
I20260926 13:42:47.201444 15479 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:47.201527 15479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:47.205950 15479 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:42:47.208640 15495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:47.210083 15479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:42:47.210245 15479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "f75e027e2f01413493dc55271df2c7bc"
format_stamp: "Formatted at 2026-09-26 13:42:45 on dist-test-slave-1wns"
server_key: "7c25c5c58448f56724689db4c7a34fdf"
server_key_iv: "97eac7c0dbb3311b3fc04d3c9c81e65e"
server_key_version: "encryptionkey@0"
I20260926 13:42:47.210927 15479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:47.231875 15479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:47.236101 15479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:47.236445 15479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:47.245198 15479 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:46133
I20260926 13:42:47.245626 15547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:46133 every 8 connection(s)
I20260926 13:42:47.246493 15479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:42:47.247370 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 15479
I20260926 13:42:47.247953 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:42311
--local_ip_for_outbound_sockets=127.1.49.1
--tserver_master_addrs=127.1.49.62:46133
--webserver_port=46023
--webserver_interface=127.1.49.1
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:42:47.252673 15548 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:42:47.255149 15548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Bootstrap starting.
I20260926 13:42:47.275295 15548 log.cc:824] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:47.286516 15548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:42:47.287011 15548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Bootstrap complete.
I20260926 13:42:47.291712 15548 raft_consensus.cc:348] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } }
I20260926 13:42:47.292296 15548 raft_consensus.cc:729] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75e027e2f01413493dc55271df2c7bc, State: Initialized, Role: FOLLOWER
I20260926 13:42:47.292975 15548 consensus_queue.cc:260] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } }
I20260926 13:42:47.293187 15548 raft_consensus.cc:388] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:42:47.293290 15548 raft_consensus.cc:482] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:42:47.293411 15548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:42:47.297098 15548 raft_consensus.cc:504] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } }
I20260926 13:42:47.297640 15548 leader_election.cc:302] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f75e027e2f01413493dc55271df2c7bc; no voters:
I20260926 13:42:47.298059 15548 leader_election.cc:288] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:42:47.298256 15552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:42:47.298593 15552 raft_consensus.cc:686] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 2 LEADER]: Becoming Leader. State: Replica: f75e027e2f01413493dc55271df2c7bc, State: Running, Role: LEADER
I20260926 13:42:47.298936 15552 consensus_queue.cc:237] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } }
I20260926 13:42:47.299677 15548 sys_catalog.cc:564] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:42:47.300414 15552 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f75e027e2f01413493dc55271df2c7bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } } }
I20260926 13:42:47.300534 15552 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:47.300733 15552 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75e027e2f01413493dc55271df2c7bc. Latest consensus state: current_term: 2 leader_uuid: "f75e027e2f01413493dc55271df2c7bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } } }
I20260926 13:42:47.300810 15552 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:47.301050 15557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:42:47.313426 15557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:42:47.315478 15557 catalog_manager.cc:1259] Loaded cluster ID: b19507baf68c4ff7a990139f79660f3d
I20260926 13:42:47.315575 15557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:42:47.316977 15557 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 13:42:47.320397 15557 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 13:42:47.326984 15557 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:42:47.328267 15557 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Loaded TSK: 0
I20260926 13:42:47.330026 15557 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:42:47.379415 15550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:47.379765 15550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:47.379892 15550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:47.384505 15550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:47.384615 15550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:47.384712 15550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:42:47.391598 15550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:42311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=46023
--tserver_master_addrs=127.1.49.62:46133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:47.392901 15550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:47.394073 15550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:47.406203 15576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:47.406203 15575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:47.406325 15578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:47.406878 15550 server_base.cc:1034] running on GCE node
I20260926 13:42:47.407413 15550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:47.408388 15550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:47.409585 15550 hybrid_clock.cc:648] HybridClock initialized: now 1790430167409567 us; error 28 us; skew 500 ppm
Sep 26 13:42:47 dist-test-slave-1wns krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:47.413277 15550 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:42:47.414601 15550 webserver.cc:460] Webserver started at http://127.1.49.1:46023/ using document root <none> and password file <none>
I20260926 13:42:47.415421 15550 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:47.415539 15550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:47.421566 15550 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:42:47.425390 15585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:47.427430 15550 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:42:47.427629 15550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "7a1e24782402468dbfb5343bd5dcc867"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "5cc8876fb6104a75b7116235e10eb378"
server_key_iv: "45647784197308a69b6b5bd7f2b81617"
server_key_version: "encryptionkey@0"
I20260926 13:42:47.428663 15550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:47.456037 15550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:47.460963 15550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:47.461251 15550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:47.462170 15550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:47.464066 15592 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:42:47.468791 15550 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:42:47.468922 15550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260926 13:42:47.469017 15550 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:42:47.471912 15550 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:42:47.472008 15550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260926 13:42:47.472458 15592 tablet_bootstrap.cc:492] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: Bootstrap starting.
I20260926 13:42:47.485122 15550 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:42311
I20260926 13:42:47.485317 15699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:42311 every 8 connection(s)
I20260926 13:42:47.486583 15550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:42:47.491497 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 15550
I20260926 13:42:47.492152 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:36147
--local_ip_for_outbound_sockets=127.1.49.2
--tserver_master_addrs=127.1.49.62:46133
--webserver_port=41621
--webserver_interface=127.1.49.2
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:42:47.497383 15592 log.cc:824] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: Log is configured to *not* fsync() on all Append() calls
Sep 26 13:42:47 dist-test-slave-1wns krb5kdc[14931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:47.508719 15592 tablet_bootstrap.cc:492] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:42:47.509174 15592 tablet_bootstrap.cc:492] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: Bootstrap complete.
I20260926 13:42:47.510635 15592 ts_tablet_manager.cc:1397] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: Time spent bootstrapping tablet: real 0.039s user 0.021s sys 0.012s
I20260926 13:42:47.511248 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.489156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:49541 (local address 127.1.49.62:46133)
0926 13:42:47.489711 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:47.489731 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:47.490214 (+ 483us) server_negotiation.cc:407] Connection header received
0926 13:42:47.491878 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:47.491902 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:47.492362 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:47.492768 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:47.495223 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.497007 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.499218 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.500131 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.500323 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:47.504408 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:47.504416 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:47.504433 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:47.504473 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:47.506714 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:47.507459 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:47.507465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:47.507471 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:47.507640 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:47.508081 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:47.508085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:47.508087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:47.508550 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:47.508666 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:47.509853 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0926 13:42:47.510255 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":150,"threads_started":1}
I20260926 13:42:47.513372 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.489173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:49541)
0926 13:42:47.490024 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:47.490084 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:47.491236 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:47.493501 (+ 2265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:47.493512 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:47.494122 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:47.494921 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:47.494952 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.497256 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.497260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:47.498995 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:47.499017 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.500718 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.500723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:47.500981 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:47.501833 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:47.501865 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:47.504135 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:47.506921 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:47.506928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:47.506944 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:47.507310 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:47.507763 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:47.507766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:47.507768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:47.507964 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:47.508751 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:47.508757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:47.509176 (+ 419us) client_negotiation.cc:769] Sending connection context
0926 13:42:47.511977 (+ 2801us) client_negotiation.cc:240] Negotiation successful
0926 13:42:47.512343 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":193,"threads_started":1}
I20260926 13:42:47.513762 15592 raft_consensus.cc:348] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } } peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } }
I20260926 13:42:47.514286 15592 raft_consensus.cc:729] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1e24782402468dbfb5343bd5dcc867, State: Initialized, Role: FOLLOWER
I20260926 13:42:47.514808 15592 consensus_queue.cc:260] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } } peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } }
I20260926 13:42:47.515959 15592 ts_tablet_manager.cc:1428] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20260926 13:42:47.516938 15700 heartbeater.cc:347] Connected to a master server at 127.1.49.62:46133
I20260926 13:42:47.517174 15700 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:47.517759 15700 heartbeater.cc:511] Master 127.1.49.62:46133 requested a full tablet report, sending...
I20260926 13:42:47.520620 15512 ts_manager.cc:194] Registered new tserver with Master: 7a1e24782402468dbfb5343bd5dcc867 (127.1.49.1:42311)
I20260926 13:42:47.524596 15512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:49541
I20260926 13:42:47.527484 15700 heartbeater.cc:503] Master 127.1.49.62:46133 was elected leader, sending a full tablet report...
Sep 26 13:42:47 dist-test-slave-1wns krb5kdc[14931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for kudu/127.1.49.1@KRBTEST.COM
I20260926 13:42:47.535751 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.523528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:42311 (local address 127.0.0.1:46656)
0926 13:42:47.523996 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:47.524057 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:47.524212 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:47.524917 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:47.524923 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:47.525368 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:47.525497 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:47.525505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.526524 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.526530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:47.527149 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:47.527158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.527872 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.528040 (+ 168us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:47.528168 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:47.528766 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:47.528778 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:47.530503 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:47.532925 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:47.532939 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:47.532945 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:47.533303 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:47.533850 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:47.533854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:47.533856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:47.533970 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:47.534888 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:47.534894 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:47.535040 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 13:42:47.535190 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 13:42:47.535544 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":126,"threads_started":1}
I20260926 13:42:47.536422 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.523632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.1.49.1:42311)
0926 13:42:47.524378 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:47.524394 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:47.524419 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:42:47.524496 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:47.524501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:47.524705 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:47.524859 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:47.525620 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.526401 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.527278 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.527732 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.527887 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:47.530658 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:47.530669 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:47.530682 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:47.530712 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:47.532777 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:47.533491 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:47.533501 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:47.533503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:47.533675 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:47.534077 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:47.534080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:47.534082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:47.534607 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:47.534803 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:47.536053 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0926 13:42:47.536241 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":95,"threads_started":1}
I20260926 13:42:47.537554 15634 tablet_service.cc:1514] Processing DeleteTablet for tablet 52485d4816cc4ca89f128b89a6af2e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:42:47 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:46656
I20260926 13:42:47.538684 15709 tablet_replica.cc:330] stopping tablet replica
I20260926 13:42:47.539026 15709 raft_consensus.cc:2229] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:42:47.539345 15709 raft_consensus.cc:2258] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:42:47.540475 15709 ts_tablet_manager.cc:1905] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:42:47.544399 15709 ts_tablet_manager.cc:1918] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:42:47.544540 15709 log.cc:1196] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal/wals/52485d4816cc4ca89f128b89a6af2e11
I20260926 13:42:47.544816 15709 ts_tablet_manager.cc:1939] T 52485d4816cc4ca89f128b89a6af2e11 P 7a1e24782402468dbfb5343bd5dcc867: Deleting consensus metadata
I20260926 13:42:47.546754 15496 catalog_manager.cc:4841] TS 7a1e24782402468dbfb5343bd5dcc867 (127.1.49.1:42311): tablet 52485d4816cc4ca89f128b89a6af2e11 (table test-table [id=fba36d0d3534452abd4eb5fc55215f19]) successfully deleted
W20260926 13:42:47.626864 15704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:47.627179 15704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:47.627228 15704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:47.631404 15704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:47.631481 15704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:47.631544 15704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:42:47.637705 15704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:36147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=41621
--tserver_master_addrs=127.1.49.62:46133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:47.638847 15704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:47.639982 15704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:47.651111 15717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:47.651161 15714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:47.651237 15715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:47.651567 15704 server_base.cc:1034] running on GCE node
I20260926 13:42:47.652207 15704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:47.652966 15704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:47.654177 15704 hybrid_clock.cc:648] HybridClock initialized: now 1790430167654139 us; error 57 us; skew 500 ppm
Sep 26 13:42:47 dist-test-slave-1wns krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:47.657501 15704 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:42:47.658526 15704 webserver.cc:460] Webserver started at http://127.1.49.2:41621/ using document root <none> and password file <none>
I20260926 13:42:47.659097 15704 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:47.659176 15704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:47.663709 15704 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:42:47.666855 15724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:47.668607 15704 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260926 13:42:47.668771 15704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "067fdeda67394d56be906073b737d5b3"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "a8a606ad51541c83926ece6ad98955c6"
server_key_iv: "6dee53dd1b72ee196245bb0f1f290e88"
server_key_version: "encryptionkey@0"
I20260926 13:42:47.669425 15704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:47.690147 15704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:47.694169 15704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:47.694406 15704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:47.695078 15704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:47.696066 15704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:42:47.696135 15704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:47.696219 15704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:42:47.696266 15704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:47.706684 15704 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:36147
I20260926 13:42:47.706727 15837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:36147 every 8 connection(s)
I20260926 13:42:47.707931 15704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:42:47.711418 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 15704
I20260926 13:42:47.711966 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:35239
--local_ip_for_outbound_sockets=127.1.49.3
--tserver_master_addrs=127.1.49.62:46133
--webserver_port=36049
--webserver_interface=127.1.49.3
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:42:47 dist-test-slave-1wns krb5kdc[14931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:47.725082 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.710132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:53745 (local address 127.1.49.62:46133)
0926 13:42:47.710328 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:47.710332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:47.711685 (+ 1353us) server_negotiation.cc:407] Connection header received
0926 13:42:47.713307 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:47.713310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:47.713374 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:47.713492 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:47.715012 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.715781 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.717239 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.717503 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.717622 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:47.720467 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:47.720471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:47.720473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:47.720504 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:47.722351 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:47.723078 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:47.723082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:47.723085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:47.723202 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:47.723622 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:47.723626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:47.723629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:47.723932 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:47.724087 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:47.724678 (+ 591us) server_negotiation.cc:299] Negotiation successful
0926 13:42:47.724873 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 13:42:47.726008 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.710572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:53745)
0926 13:42:47.711430 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:47.711500 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:47.712920 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:47.713753 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:47.713761 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:47.714164 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:47.714840 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:47.714856 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.715952 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.715960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:47.717097 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:47.717109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.717689 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.717694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:47.717878 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:47.718581 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:47.718599 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:47.720253 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:47.722523 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:47.722531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:47.722552 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:47.722949 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:47.723310 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:47.723315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:47.723317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:47.723493 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:47.724072 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:47.724081 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:47.724395 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 13:42:47.724622 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 13:42:47.725036 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":123,"threads_started":1}
I20260926 13:42:47.727516 15838 heartbeater.cc:347] Connected to a master server at 127.1.49.62:46133
I20260926 13:42:47.727892 15838 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:47.728572 15838 heartbeater.cc:511] Master 127.1.49.62:46133 requested a full tablet report, sending...
I20260926 13:42:47.730105 15512 ts_manager.cc:194] Registered new tserver with Master: 067fdeda67394d56be906073b737d5b3 (127.1.49.2:36147)
I20260926 13:42:47.730870 15512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:53745
W20260926 13:42:47.836087 15841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:47.836424 15841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:47.836521 15841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:47.840917 15841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:47.841027 15841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:47.841120 15841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:42:47.846928 15841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:35239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=36049
--tserver_master_addrs=127.1.49.62:46133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:47.848244 15841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:47.849403 15841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:47.860929 15846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:47.860932 15849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:47.860962 15847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:47.861202 15841 server_base.cc:1034] running on GCE node
I20260926 13:42:47.861683 15841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:47.862483 15841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:47.863680 15841 hybrid_clock.cc:648] HybridClock initialized: now 1790430167863664 us; error 47 us; skew 500 ppm
Sep 26 13:42:47 dist-test-slave-1wns krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:47.866874 15841 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:42:47.867966 15841 webserver.cc:460] Webserver started at http://127.1.49.3:36049/ using document root <none> and password file <none>
I20260926 13:42:47.868592 15841 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:47.868701 15841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:47.873155 15841 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 13:42:47.875787 15856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:47.877367 15841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:42:47.877527 15841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "174e702370d844e2b08c810391415346"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "4ef3859951d4517c0aaaabd316213020"
server_key_iv: "515d9c3fb6341df2ea6ac8f4ca04f264"
server_key_version: "encryptionkey@0"
I20260926 13:42:47.878183 15841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:47.919984 15841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:47.923719 15841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:47.924000 15841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:47.924701 15841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:47.925657 15841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:42:47.925733 15841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:47.925849 15841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:42:47.925899 15841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:47.936926 15841 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:35239
I20260926 13:42:47.936995 15969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:35239 every 8 connection(s)
I20260926 13:42:47.938138 15841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:42:47.940919 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 15841
Sep 26 13:42:47 dist-test-slave-1wns krb5kdc[14931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:42:47.952108 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.941702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.1.49.62:46133)
0926 13:42:47.941951 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:47.941957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:47.942161 (+ 204us) server_negotiation.cc:407] Connection header received
0926 13:42:47.942346 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:47.942348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:47.942395 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:47.942487 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:47.943326 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.943940 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.945203 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.945560 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.945686 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:47.947765 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:47.947771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:47.947775 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:47.947837 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:47.950018 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:47.950626 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:47.950630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:47.950632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:47.950753 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:47.951056 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:47.951063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:47.951065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:47.951297 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:47.951527 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:47.951701 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 13:42:47.951888 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260926 13:42:47.956393 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.940469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:32807 (local address 127.1.49.62:46133)
0926 13:42:47.940622 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:47.940626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:47.941994 (+ 1368us) server_negotiation.cc:407] Connection header received
0926 13:42:47.943050 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:47.943053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:47.943104 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:47.943193 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:47.944968 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.945809 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.947792 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.948188 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.948310 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:47.951372 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:47.951378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:47.951381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:47.951402 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:47.953640 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:47.954503 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:47.954507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:47.954508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:47.954596 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:47.955026 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:47.955029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:47.955030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:47.955320 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:47.955411 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:47.956108 (+ 697us) server_negotiation.cc:299] Negotiation successful
0926 13:42:47.956239 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:42:47.957401 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.940920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:32807)
0926 13:42:47.941807 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:47.941867 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:47.942777 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:47.943474 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:47.943484 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:47.944029 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:47.944781 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:47.944803 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.946008 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.946012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:47.947663 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:47.947679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.948380 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.948385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:47.948630 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:47.949303 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:47.949330 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:47.951170 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:47.953807 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:47.953813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:47.953831 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:47.954378 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:47.954729 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:47.954732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:47.954733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:47.954922 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:47.955453 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:47.955460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:47.955755 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 13:42:47.956024 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 13:42:47.956477 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":187,"threads_started":1}
I20260926 13:42:47.958938 15970 heartbeater.cc:347] Connected to a master server at 127.1.49.62:46133
I20260926 13:42:47.959267 15970 heartbeater.cc:464] Registering TS with master...
I20260926 13:42:47.960023 15970 heartbeater.cc:511] Master 127.1.49.62:46133 requested a full tablet report, sending...
I20260926 13:42:47.961555 15512 ts_manager.cc:194] Registered new tserver with Master: 174e702370d844e2b08c810391415346 (127.1.49.3:35239)
I20260926 13:42:47.962131 15512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:32807
I20260926 13:42:47.966538 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:42:47.971385 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.967384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.1.49.62:46133)
0926 13:42:47.967520 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:47.967524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:47.967570 (+ 46us) server_negotiation.cc:407] Connection header received
0926 13:42:47.967728 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:47.967730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:47.967929 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:47.968032 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:42:47.968251 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.968833 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.969859 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.970155 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.970256 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:47.970584 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:42:47.971034 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:42:47.971180 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:47.971206 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 13:42:47.971264 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 13:42:47.972282 15512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:42:47.975898 15512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:42:47.985541 15634 tablet_service.cc:1467] Processing CreateTablet for tablet 9c22ee5c21d643d39d076f66c4fd1575 (DEFAULT_TABLE table=test-table [id=5cf94ebfb0f64bb193f18d2db96f98c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:42:47.986304 15634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c22ee5c21d643d39d076f66c4fd1575. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:42:47.989431 15592 tablet_bootstrap.cc:492] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867: Bootstrap starting.
I20260926 13:42:47.990356 15592 tablet_bootstrap.cc:654] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867: Neither blocks nor log segments found. Creating new log.
I20260926 13:42:47.990406 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.984211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:36147 (local address 127.0.0.1:57444)
0926 13:42:47.984337 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:47.984354 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:47.984450 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:47.985456 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:47.985459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:47.985477 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:42:47.985596 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:47.985603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.987481 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.987484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:47.988678 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:47.988689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.989887 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.989891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:47.990014 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:47.990074 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 13:42:47.990138 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 13:42:47.990247 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":44}
I20260926 13:42:47.990505 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.984704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49410 (local address 127.1.49.3:35239)
0926 13:42:47.985195 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:47.985204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:47.985223 (+ 19us) server_negotiation.cc:407] Connection header received
0926 13:42:47.985288 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:47.985293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:47.985541 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:47.985703 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:42:47.986182 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.987159 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.988944 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.989876 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.989982 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:47.990159 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:47.990262 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 13:42:47.990357 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":133,"threads_started":1}
I20260926 13:42:47.990980 15980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.984509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:35239 (local address 127.0.0.1:49410)
0926 13:42:47.984778 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:47.984792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:47.984893 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:47.985909 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:47.985912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:47.985932 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:42:47.986039 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:47.986046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.987275 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.987282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:47.988826 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:47.988833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.989997 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.990001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:47.990113 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:47.990139 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 13:42:47.990231 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 13:42:47.990336 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":115,"threads_started":1}
I20260926 13:42:47.991348 15979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:47.984394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57444 (local address 127.1.49.2:36147)
0926 13:42:47.984885 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:47.984897 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:47.984923 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:42:47.985011 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:47.985016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:47.985272 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:47.986090 (+ 818us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:42:47.986274 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.987343 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.988818 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:47.989730 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:47.990703 (+ 973us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:47.990860 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:47.990912 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 13:42:47.991019 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":149,"threads_started":1}
I20260926 13:42:47.992322 15592 tablet_bootstrap.cc:492] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867: No bootstrap required, opened a new log
I20260926 13:42:47.992417 15592 ts_tablet_manager.cc:1397] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:42:47.992892 15592 raft_consensus.cc:348] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:47.993007 15592 raft_consensus.cc:374] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:42:47.993042 15592 raft_consensus.cc:729] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1e24782402468dbfb5343bd5dcc867, State: Initialized, Role: FOLLOWER
I20260926 13:42:47.993180 15592 consensus_queue.cc:260] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:47.993440 15592 ts_tablet_manager.cc:1428] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:42:47.994573 15904 tablet_service.cc:1467] Processing CreateTablet for tablet 9c22ee5c21d643d39d076f66c4fd1575 (DEFAULT_TABLE table=test-table [id=5cf94ebfb0f64bb193f18d2db96f98c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:42:47.995682 15904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c22ee5c21d643d39d076f66c4fd1575. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:42:47.998085 15772 tablet_service.cc:1467] Processing CreateTablet for tablet 9c22ee5c21d643d39d076f66c4fd1575 (DEFAULT_TABLE table=test-table [id=5cf94ebfb0f64bb193f18d2db96f98c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:42:47.999481 15772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c22ee5c21d643d39d076f66c4fd1575. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:42:48.004118 15983 tablet_bootstrap.cc:492] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346: Bootstrap starting.
I20260926 13:42:48.006448 15983 tablet_bootstrap.cc:654] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346: Neither blocks nor log segments found. Creating new log.
I20260926 13:42:48.007670 15983 log.cc:824] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:48.007915 15984 tablet_bootstrap.cc:492] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3: Bootstrap starting.
I20260926 13:42:48.010149 15983 tablet_bootstrap.cc:492] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346: No bootstrap required, opened a new log
I20260926 13:42:48.010387 15983 ts_tablet_manager.cc:1397] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:42:48.010890 15984 tablet_bootstrap.cc:654] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3: Neither blocks nor log segments found. Creating new log.
I20260926 13:42:48.011880 15984 log.cc:824] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:48.014344 15984 tablet_bootstrap.cc:492] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3: No bootstrap required, opened a new log
I20260926 13:42:48.014588 15984 ts_tablet_manager.cc:1397] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 13:42:48.014798 15983 raft_consensus.cc:348] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:48.015136 15983 raft_consensus.cc:374] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:42:48.015205 15983 raft_consensus.cc:729] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174e702370d844e2b08c810391415346, State: Initialized, Role: FOLLOWER
I20260926 13:42:48.015753 15983 consensus_queue.cc:260] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:48.016638 15970 heartbeater.cc:503] Master 127.1.49.62:46133 was elected leader, sending a full tablet report...
I20260926 13:42:48.017023 15983 ts_tablet_manager.cc:1428] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 13:42:48.018704 15984 raft_consensus.cc:348] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:48.019066 15984 raft_consensus.cc:374] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:42:48.019135 15984 raft_consensus.cc:729] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067fdeda67394d56be906073b737d5b3, State: Initialized, Role: FOLLOWER
I20260926 13:42:48.019627 15984 consensus_queue.cc:260] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:48.020447 15838 heartbeater.cc:503] Master 127.1.49.62:46133 was elected leader, sending a full tablet report...
I20260926 13:42:48.020884 15984 ts_tablet_manager.cc:1428] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 13:42:48.129326 15988 raft_consensus.cc:482] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:42:48.129598 15988 raft_consensus.cc:504] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:48.130988 15988 leader_election.cc:288] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 174e702370d844e2b08c810391415346 (127.1.49.3:35239), 7a1e24782402468dbfb5343bd5dcc867 (127.1.49.1:42311)
I20260926 13:42:48.135353 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:48.131687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:41945 (local address 127.1.49.3:35239)
0926 13:42:48.131883 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:48.131890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:48.131911 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:42:48.131977 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:48.131980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:48.132025 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:48.132125 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:42:48.132449 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:48.133019 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:48.134204 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:48.134712 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:48.134852 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:48.134925 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:48.135122 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 13:42:48.135210 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 13:42:48.135353 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:48.131448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:35239 (local address 127.1.49.2:41945)
0926 13:42:48.131649 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:48.131667 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:48.131804 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:48.132139 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:48.132142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:48.132170 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:42:48.132318 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:48.132324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:48.133149 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:48.133162 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:48.134056 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:48.134066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:48.134857 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:48.134861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:48.134986 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:48.135005 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 13:42:48.135101 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 13:42:48.135197 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":1}
I20260926 13:42:48.136096 15924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c22ee5c21d643d39d076f66c4fd1575" candidate_uuid: "067fdeda67394d56be906073b737d5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174e702370d844e2b08c810391415346" is_pre_election: true
I20260926 13:42:48.136471 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:48.131835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:42311 (local address 127.1.49.2:34207)
0926 13:42:48.132317 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:48.132334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:48.132473 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:48.132798 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:48.132801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:48.132818 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:42:48.132966 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:48.132972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:48.133748 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:48.133754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:48.134690 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:48.134700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:48.136002 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:48.136005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:48.136091 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:48.136110 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 13:42:48.136185 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 13:42:48.136277 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":177,"threads_started":1}
I20260926 13:42:48.136492 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:48.132009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:34207 (local address 127.1.49.1:42311)
0926 13:42:48.132319 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:48.132325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:48.132429 (+ 104us) server_negotiation.cc:407] Connection header received
0926 13:42:48.132597 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:48.132606 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:48.132665 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:48.132790 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:42:48.133086 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:48.133620 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:48.134836 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:48.135473 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:48.135680 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:48.135929 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:48.136234 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 13:42:48.136345 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":83,"threads_started":1}
I20260926 13:42:48.136520 15924 raft_consensus.cc:2454] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 067fdeda67394d56be906073b737d5b3 in term 0.
I20260926 13:42:48.137070 15725 leader_election.cc:302] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067fdeda67394d56be906073b737d5b3, 174e702370d844e2b08c810391415346; no voters:
I20260926 13:42:48.137399 15988 raft_consensus.cc:2781] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:42:48.137367 15652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c22ee5c21d643d39d076f66c4fd1575" candidate_uuid: "067fdeda67394d56be906073b737d5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a1e24782402468dbfb5343bd5dcc867" is_pre_election: true
I20260926 13:42:48.137490 15988 raft_consensus.cc:482] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:42:48.137526 15988 raft_consensus.cc:3037] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:48.137706 15652 raft_consensus.cc:2454] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 067fdeda67394d56be906073b737d5b3 in term 0.
I20260926 13:42:48.139011 15988 raft_consensus.cc:504] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:48.139451 15988 leader_election.cc:288] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [CANDIDATE]: Term 1 election: Requested vote from peers 174e702370d844e2b08c810391415346 (127.1.49.3:35239), 7a1e24782402468dbfb5343bd5dcc867 (127.1.49.1:42311)
I20260926 13:42:48.139952 15652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c22ee5c21d643d39d076f66c4fd1575" candidate_uuid: "067fdeda67394d56be906073b737d5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a1e24782402468dbfb5343bd5dcc867"
I20260926 13:42:48.139953 15924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c22ee5c21d643d39d076f66c4fd1575" candidate_uuid: "067fdeda67394d56be906073b737d5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174e702370d844e2b08c810391415346"
I20260926 13:42:48.140117 15652 raft_consensus.cc:3037] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:48.140110 15924 raft_consensus.cc:3037] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:42:48.141539 15924 raft_consensus.cc:2454] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 067fdeda67394d56be906073b737d5b3 in term 1.
I20260926 13:42:48.141539 15652 raft_consensus.cc:2454] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 067fdeda67394d56be906073b737d5b3 in term 1.
I20260926 13:42:48.141901 15725 leader_election.cc:302] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067fdeda67394d56be906073b737d5b3, 174e702370d844e2b08c810391415346; no voters:
I20260926 13:42:48.142403 15988 raft_consensus.cc:2781] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:42:48.142659 15988 raft_consensus.cc:686] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [term 1 LEADER]: Becoming Leader. State: Replica: 067fdeda67394d56be906073b737d5b3, State: Running, Role: LEADER
I20260926 13:42:48.143100 15988 consensus_queue.cc:237] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:48.146535 15512 catalog_manager.cc:5495] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 067fdeda67394d56be906073b737d5b3 (127.1.49.2). New cstate: current_term: 1 leader_uuid: "067fdeda67394d56be906073b737d5b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } health_report { overall_health: UNKNOWN } } }
W20260926 13:42:48.189251 15971 tablet.cc:2369] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:42:48.190244 15979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:48.185575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57452 (local address 127.1.49.2:36147)
0926 13:42:48.185758 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:48.185762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:48.185786 (+ 24us) server_negotiation.cc:407] Connection header received
0926 13:42:48.185995 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:48.185998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:48.186076 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:48.186186 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:42:48.186498 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:48.187099 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:48.188390 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:48.188641 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:48.188748 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:48.189094 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:42:48.189703 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:42:48.189841 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:48.189957 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 13:42:48.190124 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 13:42:48.197278 15652 raft_consensus.cc:1261] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Refusing update from remote peer 067fdeda67394d56be906073b737d5b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:42:48.197315 15924 raft_consensus.cc:1261] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346 [term 1 FOLLOWER]: Refusing update from remote peer 067fdeda67394d56be906073b737d5b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:42:48.198173 15988 consensus_queue.cc:1035] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:42:48.198500 15991 consensus_queue.cc:1035] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:42:48.207038 15994 mvcc.cc:205] Tried to move back new op lower bound from 7333601968926470144 to 7333601968717180928. Current Snapshot: MvccSnapshot[applied={T|T < 7333601968926470144}]
I20260926 13:42:48.218284 15512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:42:48.218474 15512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:42:48.221333 15512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Sending DeleteTablet for 3 replicas of tablet 9c22ee5c21d643d39d076f66c4fd1575
I20260926 13:42:48.222327 15772 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c22ee5c21d643d39d076f66c4fd1575 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:42:48 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:57444
I20260926 13:42:48.222337 15904 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c22ee5c21d643d39d076f66c4fd1575 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:42:48 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:49410
I20260926 13:42:48.222767 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 15550
I20260926 13:42:48.222899 15634 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c22ee5c21d643d39d076f66c4fd1575 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:42:48 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:46656
I20260926 13:42:48.223634 16003 tablet_replica.cc:330] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3: stopping tablet replica
I20260926 13:42:48.224056 16003 raft_consensus.cc:2229] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:42:48.224498 16003 raft_consensus.cc:2258] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:42:48.225023 16004 tablet_replica.cc:330] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346: stopping tablet replica
I20260926 13:42:48.225396 16004 raft_consensus.cc:2229] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:42:48.225800 16004 raft_consensus.cc:2258] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:42:48.226389 16003 ts_tablet_manager.cc:1905] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:42:48.229880 16004 ts_tablet_manager.cc:1905] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:42:48.230628 16003 ts_tablet_manager.cc:1918] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:42:48.230768 16003 log.cc:1196] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/9c22ee5c21d643d39d076f66c4fd1575
I20260926 13:42:48.231189 16003 ts_tablet_manager.cc:1939] T 9c22ee5c21d643d39d076f66c4fd1575 P 067fdeda67394d56be906073b737d5b3: Deleting consensus metadata
W20260926 13:42:48.232244 15496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:42311 (error 108)
I20260926 13:42:48.233305 15496 catalog_manager.cc:4841] TS 067fdeda67394d56be906073b737d5b3 (127.1.49.2:36147): tablet 9c22ee5c21d643d39d076f66c4fd1575 (table test-table [id=5cf94ebfb0f64bb193f18d2db96f98c4]) successfully deleted
I20260926 13:42:48.233587 16004 ts_tablet_manager.cc:1918] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:42:48.233790 16004 log.cc:1196] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/9c22ee5c21d643d39d076f66c4fd1575
W20260926 13:42:48.233811 15980 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:42:48.233120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:42311 (local address 127.0.0.1:46658)
0926 13:42:48.233267 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:48.233285 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:48.233386 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:48.233684 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:42311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 13:42:48.234323 16004 ts_tablet_manager.cc:1939] T 9c22ee5c21d643d39d076f66c4fd1575 P 174e702370d844e2b08c810391415346: Deleting consensus metadata
W20260926 13:42:48.233996 15496 catalog_manager.cc:4584] TS 7a1e24782402468dbfb5343bd5dcc867 (127.1.49.1:42311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c22ee5c21d643d39d076f66c4fd1575: Network error: Client connection negotiation failed: client connection to 127.1.49.1:42311: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:42:48.235009 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 15704
I20260926 13:42:48.237773 15496 catalog_manager.cc:4841] TS 174e702370d844e2b08c810391415346 (127.1.49.3:35239): tablet 9c22ee5c21d643d39d076f66c4fd1575 (table test-table [id=5cf94ebfb0f64bb193f18d2db96f98c4]) successfully deleted
I20260926 13:42:48.241425 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 15841
I20260926 13:42:48.248070 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 15479
I20260926 13:42:48.256083 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:46133
--webserver_interface=127.1.49.62
--webserver_port=36739
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:46133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:42:48.384701 16006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:48.385347 16006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:48.385536 16006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:48.392603 16006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:42:48.392756 16006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:48.392795 16006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:48.392827 16006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:42:48.392866 16006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:42:48.399405 16006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:46133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:46133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=36739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:48.400719 16006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:48.401862 16006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:48.411685 16011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:48.411674 16014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:48.411674 16012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:48.412184 16006 server_base.cc:1034] running on GCE node
I20260926 13:42:48.413182 16006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:48.414587 16006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:48.416030 16006 hybrid_clock.cc:648] HybridClock initialized: now 1790430168415996 us; error 53 us; skew 500 ppm
Sep 26 13:42:48 dist-test-slave-1wns krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430168, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:48.420182 16006 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:42:48.421293 16006 webserver.cc:460] Webserver started at http://127.1.49.62:36739/ using document root <none> and password file <none>
I20260926 13:42:48.421924 16006 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:48.422003 16006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:48.426491 16006 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:42:48.429306 16021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:48.431030 16006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:42:48.431195 16006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "f75e027e2f01413493dc55271df2c7bc"
format_stamp: "Formatted at 2026-09-26 13:42:45 on dist-test-slave-1wns"
server_key: "7c25c5c58448f56724689db4c7a34fdf"
server_key_iv: "97eac7c0dbb3311b3fc04d3c9c81e65e"
server_key_version: "encryptionkey@0"
I20260926 13:42:48.431902 16006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:48.450305 16006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:48.454492 16006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:48.454677 16006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:48.462777 16006 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:46133
I20260926 13:42:48.463219 16073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:46133 every 8 connection(s)
I20260926 13:42:48.464219 16006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:42:48.469833 16074 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:42:48.471957 16074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Bootstrap starting.
I20260926 13:42:48.474485 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 16006
I20260926 13:42:48.474959 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:42311
--local_ip_for_outbound_sockets=127.1.49.1
--tserver_master_addrs=127.1.49.62:46133
--webserver_port=46023
--webserver_interface=127.1.49.1
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:42:48.491412 16074 log.cc:824] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:48.503350 16074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:42:48.503849 16074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc: Bootstrap complete.
I20260926 13:42:48.508361 16074 raft_consensus.cc:348] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } }
I20260926 13:42:48.508893 16074 raft_consensus.cc:729] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75e027e2f01413493dc55271df2c7bc, State: Initialized, Role: FOLLOWER
I20260926 13:42:48.509553 16074 consensus_queue.cc:260] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } }
I20260926 13:42:48.509714 16074 raft_consensus.cc:388] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:42:48.509805 16074 raft_consensus.cc:482] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:42:48.509931 16074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 2 FOLLOWER]: Advancing to term 3
I20260926 13:42:48.512243 16074 raft_consensus.cc:504] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } }
I20260926 13:42:48.512686 16074 leader_election.cc:302] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f75e027e2f01413493dc55271df2c7bc; no voters:
I20260926 13:42:48.513088 16074 leader_election.cc:288] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 13:42:48.513192 16078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 3 FOLLOWER]: Leader election won for term 3
I20260926 13:42:48.513391 16078 raft_consensus.cc:686] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [term 3 LEADER]: Becoming Leader. State: Replica: f75e027e2f01413493dc55271df2c7bc, State: Running, Role: LEADER
I20260926 13:42:48.513702 16078 consensus_queue.cc:237] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } }
I20260926 13:42:48.514487 16074 sys_catalog.cc:564] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:42:48.514706 16078 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75e027e2f01413493dc55271df2c7bc. Latest consensus state: current_term: 3 leader_uuid: "f75e027e2f01413493dc55271df2c7bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } } }
I20260926 13:42:48.514829 16078 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:48.515202 16082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:42:48.515350 16079 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f75e027e2f01413493dc55271df2c7bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e027e2f01413493dc55271df2c7bc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 46133 } } }
I20260926 13:42:48.515539 16079 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75e027e2f01413493dc55271df2c7bc [sys.catalog]: This master's current role is: LEADER
I20260926 13:42:48.521036 16082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:42:48.521739 16082 catalog_manager.cc:1259] Loaded cluster ID: b19507baf68c4ff7a990139f79660f3d
I20260926 13:42:48.521803 16082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 13:42:48.522861 16082 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8fe37279ad google::LogMessage::Fail() at ??:0
@ 0x7f8fe372a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f8fe37274de google::LogMessage::Flush() at ??:0
@ 0x7f8fe3728a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8fee06afdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f8fee06bd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8fee068579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f8fee0ac640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8fe59793d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8fe5979c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f8fe597b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8fe596a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8fe6fae6db start_thread at ??:0
@ 0x7f8fe222561f clone at ??:0
W20260926 13:42:48.612551 16076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:48.612883 16076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:48.612985 16076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:48.617311 16076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:48.617444 16076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:48.617551 16076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:42:48.623540 16076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:42311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=46023
--tserver_master_addrs=127.1.49.62:46133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:48.624769 16076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:48.625877 16076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:48.636313 16116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:48.636282 16115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:48.636567 16076 server_base.cc:1034] running on GCE node
W20260926 13:42:48.636668 16118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:48.637109 16076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:48.637933 16076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:48.639232 16076 hybrid_clock.cc:648] HybridClock initialized: now 1790430168639211 us; error 39 us; skew 500 ppm
Sep 26 13:42:48 dist-test-slave-1wns krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430168, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:48.642419 16076 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:42:48.643483 16076 webserver.cc:460] Webserver started at http://127.1.49.1:46023/ using document root <none> and password file <none>
I20260926 13:42:48.644169 16076 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:48.644239 16076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:48.648756 16076 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 13:42:48.651367 16125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:48.653045 16076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 13:42:48.653236 16076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "7a1e24782402468dbfb5343bd5dcc867"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "5cc8876fb6104a75b7116235e10eb378"
server_key_iv: "45647784197308a69b6b5bd7f2b81617"
server_key_version: "encryptionkey@0"
I20260926 13:42:48.653925 16076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:48.681324 16076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:48.685195 16076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:48.685417 16076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:48.686064 16076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:48.687279 16132 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:42:48.693311 16076 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:42:48.693442 16076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 13:42:48.693540 16076 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:42:48.695545 16076 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:42:48.695639 16076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 13:42:48.696003 16132 tablet_bootstrap.cc:492] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867: Bootstrap starting.
I20260926 13:42:48.707885 16076 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:42311
I20260926 13:42:48.708249 16239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:42311 every 8 connection(s)
I20260926 13:42:48.709233 16076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:42:48.712852 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 16076
I20260926 13:42:48.713336 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:36147
--local_ip_for_outbound_sockets=127.1.49.2
--tserver_master_addrs=127.1.49.62:46133
--webserver_port=41621
--webserver_interface=127.1.49.2
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:42:48.719329 16132 log.cc:824] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867: Log is configured to *not* fsync() on all Append() calls
I20260926 13:42:48.729749 16132 tablet_bootstrap.cc:492] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:42:48.730289 16132 tablet_bootstrap.cc:492] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867: Bootstrap complete.
I20260926 13:42:48.731896 16132 ts_tablet_manager.cc:1397] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867: Time spent bootstrapping tablet: real 0.036s user 0.032s sys 0.003s
I20260926 13:42:48.735622 16132 raft_consensus.cc:348] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:48.736321 16132 raft_consensus.cc:729] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1e24782402468dbfb5343bd5dcc867, State: Initialized, Role: FOLLOWER
I20260926 13:42:48.736999 16132 consensus_queue.cc:260] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:48.738230 16132 ts_tablet_manager.cc:1428] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260926 13:42:48.768929 16242 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:42:48.711407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:56641)
0926 13:42:48.711959 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:48.712019 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:48.713897 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:48.767929 (+ 54032us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":153,"threads_started":1}
W20260926 13:42:48.769639 16240 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:46133 (0 consecutive failures): Network error: Failed to ping master at 127.1.49.62:46133: Client connection negotiation failed: client connection to 127.1.49.62:46133: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:42:48.770417 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:48.770126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:47273)
0926 13:42:48.770264 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:48.770344 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 13:42:48.771095 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:48.770870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:58599)
0926 13:42:48.770974 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:48.771024 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 13:42:48.771296 16240 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:42:48.837664 16243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:48.837997 16243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:48.838048 16243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:48.842830 16243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:48.842923 16243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:48.843056 16243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:42:48.849130 16243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:36147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=41621
--tserver_master_addrs=127.1.49.62:46133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:48.850318 16243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:48.851380 16243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:48.862915 16251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:48.862907 16250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:48.862874 16253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:48.863054 16243 server_base.cc:1034] running on GCE node
I20260926 13:42:48.863900 16243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:48.864732 16243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:48.865950 16243 hybrid_clock.cc:648] HybridClock initialized: now 1790430168865872 us; error 95 us; skew 500 ppm
Sep 26 13:42:48 dist-test-slave-1wns krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430168, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:48.869215 16243 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:42:48.870208 16243 webserver.cc:460] Webserver started at http://127.1.49.2:41621/ using document root <none> and password file <none>
I20260926 13:42:48.870810 16243 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:48.870903 16243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:48.875268 16243 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:42:48.877723 16260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:48.879112 16243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:42:48.879256 16243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "067fdeda67394d56be906073b737d5b3"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "a8a606ad51541c83926ece6ad98955c6"
server_key_iv: "6dee53dd1b72ee196245bb0f1f290e88"
server_key_version: "encryptionkey@0"
I20260926 13:42:48.880016 16243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:48.892244 16243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:48.896597 16243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:48.896849 16243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:48.897540 16243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:48.898497 16243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:42:48.898573 16243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:48.898646 16243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:42:48.898689 16243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:48.909343 16243 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:36147
I20260926 13:42:48.909360 16373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:36147 every 8 connection(s)
I20260926 13:42:48.910578 16243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:42:48.914450 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 16243
I20260926 13:42:48.914886 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:35239
--local_ip_for_outbound_sockets=127.1.49.3
--tserver_master_addrs=127.1.49.62:46133
--webserver_port=36049
--webserver_interface=127.1.49.3
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:42:48.915400 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:48.913371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:47943)
0926 13:42:48.914168 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:48.914295 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":139,"threads_started":1}
W20260926 13:42:48.916031 16374 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:46133 (0 consecutive failures): Network error: Failed to ping master at 127.1.49.62:46133: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
I20260926 13:42:48.916961 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:48.916631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:50987)
0926 13:42:48.916794 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:48.916871 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 13:42:48.917794 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:48.917574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:49869)
0926 13:42:48.917687 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:48.917738 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 13:42:48.917979 16374 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:42:49.037166 16377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:42:49.037521 16377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:42:49.037599 16377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:42:49.041922 16377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:42:49.042042 16377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:42:49.042137 16377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:42:49.048444 16377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:35239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=36049
--tserver_master_addrs=127.1.49.62:46133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:42:49.049798 16377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:42:49.050920 16377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:42:49.062146 16382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:49.062364 16377 server_base.cc:1034] running on GCE node
W20260926 13:42:49.062131 16383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:42:49.062187 16385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:42:49.063015 16377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:42:49.063788 16377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:42:49.065043 16377 hybrid_clock.cc:648] HybridClock initialized: now 1790430169065022 us; error 34 us; skew 500 ppm
Sep 26 13:42:49 dist-test-slave-1wns krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430169, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:42:49.068143 16377 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:42:49.069159 16377 webserver.cc:460] Webserver started at http://127.1.49.3:36049/ using document root <none> and password file <none>
I20260926 13:42:49.069744 16377 fs_manager.cc:359] Metadata directory not provided
I20260926 13:42:49.069809 16377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:42:49.074165 16377 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 13:42:49.076539 16392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:49.078059 16377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:42:49.078243 16377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "174e702370d844e2b08c810391415346"
format_stamp: "Formatted at 2026-09-26 13:42:46 on dist-test-slave-1wns"
server_key: "4ef3859951d4517c0aaaabd316213020"
server_key_iv: "515d9c3fb6341df2ea6ac8f4ca04f264"
server_key_version: "encryptionkey@0"
I20260926 13:42:49.078878 16377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:42:49.097146 16377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:42:49.101145 16377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:42:49.101325 16377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:42:49.101948 16377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:42:49.102897 16377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:42:49.102950 16377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:49.103019 16377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:42:49.103045 16377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:42:49.113821 16377 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:35239
I20260926 13:42:49.113842 16505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:35239 every 8 connection(s)
I20260926 13:42:49.115041 16377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:42:49.119318 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:49.117610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:45373)
0926 13:42:49.118430 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:49.118550 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":129,"threads_started":1}
W20260926 13:42:49.120004 16506 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:46133 (0 consecutive failures): Network error: Failed to ping master at 127.1.49.62:46133: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
I20260926 13:42:49.120713 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:49.120451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:47499)
0926 13:42:49.120594 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:49.120648 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 13:42:49.121407 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:49.121207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:54267)
0926 13:42:49.121312 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:49.121355 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 13:42:49.121622 16506 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 13:42:49.123333 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 16377
I20260926 13:42:49.772775 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:49.772251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:60065)
0926 13:42:49.772547 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:49.772617 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":81,"threads_started":1}
I20260926 13:42:49.919193 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:49.918682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:40739)
0926 13:42:49.919005 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:49.919073 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":69,"threads_started":1}
I20260926 13:42:50.122809 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:50.122282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:59897)
0926 13:42:50.122611 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:50.122689 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":76,"threads_started":1}
I20260926 13:42:50.174942 16513 raft_consensus.cc:482] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:42:50.175164 16513 raft_consensus.cc:504] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:50.176513 16513 leader_election.cc:288] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 067fdeda67394d56be906073b737d5b3 (127.1.49.2:36147), 174e702370d844e2b08c810391415346 (127.1.49.3:35239)
Sep 26 13:42:50 dist-test-slave-1wns krb5kdc[14931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430168, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.2@KRBTEST.COM
Sep 26 13:42:50 dist-test-slave-1wns krb5kdc[14931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430168, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.3@KRBTEST.COM
I20260926 13:42:50.192591 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:50.176939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:44705 (local address 127.1.49.2:36147)
0926 13:42:50.177459 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:50.177488 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:50.177537 (+ 49us) server_negotiation.cc:407] Connection header received
0926 13:42:50.178276 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:50.178282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:50.178660 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:50.178981 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:50.180504 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:50.181568 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:50.183111 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:50.183876 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:50.184057 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:50.187098 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:50.187105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:50.187120 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:50.187153 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:50.189648 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:50.190282 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:50.190290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:50.190295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:50.190433 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:50.190839 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:50.190843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:50.190845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:50.191312 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:50.191462 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:50.192095 (+ 633us) server_negotiation.cc:299] Negotiation successful
0926 13:42:50.192368 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":155,"threads_started":1}
I20260926 13:42:50.192620 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:50.176763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:36147 (local address 127.1.49.1:44705)
0926 13:42:50.176948 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:50.176965 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:50.177068 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:50.179047 (+ 1979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:50.179053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:50.179639 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:50.180311 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:50.180333 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:50.181738 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:50.181741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:50.182969 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:50.182981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:50.184010 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:50.184014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:50.184303 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:50.185085 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:50.185097 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:50.186803 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:50.189810 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:50.189816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:50.189833 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:50.190122 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:50.190585 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:50.190588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:50.190590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:50.190740 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:50.191446 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:50.191453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:50.191722 (+ 269us) client_negotiation.cc:769] Sending connection context
0926 13:42:50.191969 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 13:42:50.192366 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":251,"spinlock_wait_cycles":1920}
I20260926 13:42:50.192677 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:50.177258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:54033 (local address 127.1.49.3:35239)
0926 13:42:50.177569 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 13:42:50.177589 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 13:42:50.177623 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:42:50.178315 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:42:50.178321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:42:50.178660 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:42:50.179008 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:42:50.180513 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:50.181568 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:50.183152 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:42:50.183914 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:42:50.184160 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:50.187435 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:42:50.187441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:42:50.187454 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:42:50.187488 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:42:50.189919 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:50.190448 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:50.190453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:50.190457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:50.190587 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:42:50.190899 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:42:50.190904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:42:50.190906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:42:50.191335 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:42:50.191486 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 13:42:50.192143 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 13:42:50.192456 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":113,"threads_started":1}
I20260926 13:42:50.193585 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:50.177036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:35239 (local address 127.1.49.1:54033)
0926 13:42:50.177430 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:50.177445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:42:50.177541 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:42:50.179049 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:42:50.179053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:42:50.179632 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:42:50.180312 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:50.180333 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:50.181739 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:50.181743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:50.182971 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:42:50.182982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:42:50.184048 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:42:50.184052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:42:50.184306 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:42:50.184908 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:42:50.184927 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:42:50.187191 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:42:50.190061 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:50.190066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:50.190069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:50.190304 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:50.190696 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:42:50.190701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:42:50.190703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:42:50.190799 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:42:50.191452 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:42:50.191456 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:42:50.191733 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 13:42:50.191969 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 13:42:50.192459 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":377,"spinlock_wait_cycles":39424,"thread_start_us":127,"threads_started":1}
W20260926 13:42:50.195204 16126 leader_election.cc:341] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 067fdeda67394d56be906073b737d5b3 (127.1.49.2:36147): Not found: Tablet not found: 9c22ee5c21d643d39d076f66c4fd1575
W20260926 13:42:50.195655 16126 leader_election.cc:341] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 174e702370d844e2b08c810391415346 (127.1.49.3:35239): Not found: Tablet not found: 9c22ee5c21d643d39d076f66c4fd1575
I20260926 13:42:50.195751 16126 leader_election.cc:302] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a1e24782402468dbfb5343bd5dcc867; no voters: 067fdeda67394d56be906073b737d5b3, 174e702370d844e2b08c810391415346
I20260926 13:42:50.196131 16513 raft_consensus.cc:2726] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:42:50.774116 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:50.773608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:42837)
0926 13:42:50.773944 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:50.774011 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":101,"threads_started":1}
I20260926 13:42:50.920543 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:50.920057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:43903)
0926 13:42:50.920365 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:50.920430 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":80,"threads_started":1}
I20260926 13:42:51.124301 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:51.123699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:35027)
0926 13:42:51.124048 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:51.124115 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":133,"threads_started":1}
I20260926 13:42:51.775689 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:51.775201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:40569)
0926 13:42:51.775509 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:51.775577 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
I20260926 13:42:51.922194 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:51.921621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:37191)
0926 13:42:51.922004 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:51.922075 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":94,"threads_started":1}
I20260926 13:42:52.125779 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:52.125208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:33975)
0926 13:42:52.125583 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:52.125655 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":84,"threads_started":1}
I20260926 13:42:52.169404 16523 raft_consensus.cc:482] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:42:52.169543 16523 raft_consensus.cc:504] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:52.170017 16523 leader_election.cc:288] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 067fdeda67394d56be906073b737d5b3 (127.1.49.2:36147), 174e702370d844e2b08c810391415346 (127.1.49.3:35239)
W20260926 13:42:52.171036 16126 leader_election.cc:341] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 067fdeda67394d56be906073b737d5b3 (127.1.49.2:36147): Not found: Tablet not found: 9c22ee5c21d643d39d076f66c4fd1575
W20260926 13:42:52.171270 16126 leader_election.cc:341] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 174e702370d844e2b08c810391415346 (127.1.49.3:35239): Not found: Tablet not found: 9c22ee5c21d643d39d076f66c4fd1575
I20260926 13:42:52.171342 16126 leader_election.cc:302] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a1e24782402468dbfb5343bd5dcc867; no voters: 067fdeda67394d56be906073b737d5b3, 174e702370d844e2b08c810391415346
I20260926 13:42:52.171497 16523 raft_consensus.cc:2726] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:42:52.777064 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:52.776643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:55239)
0926 13:42:52.776911 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:52.776966 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260926 13:42:52.923648 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:52.923100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:35561)
0926 13:42:52.923476 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:52.923539 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":156,"threads_started":1}
I20260926 13:42:53.127394 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:53.126835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:35771)
0926 13:42:53.127206 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:53.127275 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":83,"threads_started":1}
I20260926 13:42:53.778941 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:53.778403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:55563)
0926 13:42:53.778769 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:53.778838 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":107,"threads_started":1}
I20260926 13:42:53.925148 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:53.924669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:48895)
0926 13:42:53.924978 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:53.925045 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":84,"threads_started":1}
I20260926 13:42:54.129004 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:54.128452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:34393)
0926 13:42:54.128761 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:54.128829 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20260926 13:42:54.780449 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:54.779993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:50401)
0926 13:42:54.780287 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:54.780350 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":97,"threads_started":1}
I20260926 13:42:54.926600 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:54.926130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:48849)
0926 13:42:54.926419 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:54.926489 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260926 13:42:55.130427 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:55.129936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:52791)
0926 13:42:55.130252 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:55.130322 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260926 13:42:55.225404 16533 raft_consensus.cc:482] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:42:55.225544 16533 raft_consensus.cc:504] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:55.225915 16533 leader_election.cc:288] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 067fdeda67394d56be906073b737d5b3 (127.1.49.2:36147), 174e702370d844e2b08c810391415346 (127.1.49.3:35239)
W20260926 13:42:55.226941 16126 leader_election.cc:341] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 174e702370d844e2b08c810391415346 (127.1.49.3:35239): Not found: Tablet not found: 9c22ee5c21d643d39d076f66c4fd1575
W20260926 13:42:55.227193 16126 leader_election.cc:341] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 067fdeda67394d56be906073b737d5b3 (127.1.49.2:36147): Not found: Tablet not found: 9c22ee5c21d643d39d076f66c4fd1575
I20260926 13:42:55.227276 16126 leader_election.cc:302] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a1e24782402468dbfb5343bd5dcc867; no voters: 067fdeda67394d56be906073b737d5b3, 174e702370d844e2b08c810391415346
I20260926 13:42:55.227499 16533 raft_consensus.cc:2726] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:42:55.782053 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:55.781515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:56875)
0926 13:42:55.781864 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:55.781941 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":123,"threads_started":1}
I20260926 13:42:55.928221 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:55.927619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:49469)
0926 13:42:55.928038 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:55.928116 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":85,"threads_started":1}
I20260926 13:42:56.132125 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:56.131509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:52069)
0926 13:42:56.131954 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:56.132012 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":167,"threads_started":1}
I20260926 13:42:56.783329 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:56.782824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:32857)
0926 13:42:56.783093 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:56.783149 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
I20260926 13:42:56.929558 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:56.929093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:47593)
0926 13:42:56.929392 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:56.929458 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
I20260926 13:42:57.133596 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:57.133053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:37933)
0926 13:42:57.133415 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:57.133478 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":151,"threads_started":1}
I20260926 13:42:57.784938 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:57.784402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:55949)
0926 13:42:57.784760 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:57.784830 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":100,"threads_started":1}
I20260926 13:42:57.931051 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:57.930518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:56837)
0926 13:42:57.930821 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:57.930887 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
I20260926 13:42:58.134876 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:58.134435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:56721)
0926 13:42:58.134722 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:58.134779 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260926 13:42:58.786620 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:58.786123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:50885)
0926 13:42:58.786405 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:58.786505 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":86,"threads_started":1}
I20260926 13:42:58.932543 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:58.932080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:47155)
0926 13:42:58.932369 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:58.932439 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":83,"threads_started":1}
I20260926 13:42:59.136689 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:59.136030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:53411)
0926 13:42:59.136475 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:59.136547 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":116,"threads_started":1}
I20260926 13:42:59.788080 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:59.787564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:41227)
0926 13:42:59.787911 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:59.787979 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260926 13:42:59.811034 16547 raft_consensus.cc:482] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:42:59.811241 16547 raft_consensus.cc:504] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067fdeda67394d56be906073b737d5b3" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36147 } } peers { permanent_uuid: "174e702370d844e2b08c810391415346" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 35239 } } peers { permanent_uuid: "7a1e24782402468dbfb5343bd5dcc867" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 42311 } }
I20260926 13:42:59.811630 16547 leader_election.cc:288] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 067fdeda67394d56be906073b737d5b3 (127.1.49.2:36147), 174e702370d844e2b08c810391415346 (127.1.49.3:35239)
W20260926 13:42:59.812695 16126 leader_election.cc:341] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 174e702370d844e2b08c810391415346 (127.1.49.3:35239): Not found: Tablet not found: 9c22ee5c21d643d39d076f66c4fd1575
W20260926 13:42:59.812860 16126 leader_election.cc:341] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 067fdeda67394d56be906073b737d5b3 (127.1.49.2:36147): Not found: Tablet not found: 9c22ee5c21d643d39d076f66c4fd1575
I20260926 13:42:59.812919 16126 leader_election.cc:302] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a1e24782402468dbfb5343bd5dcc867; no voters: 067fdeda67394d56be906073b737d5b3, 174e702370d844e2b08c810391415346
I20260926 13:42:59.813119 16547 raft_consensus.cc:2726] T 9c22ee5c21d643d39d076f66c4fd1575 P 7a1e24782402468dbfb5343bd5dcc867 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:42:59.934055 16548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:42:59.933548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:48841)
0926 13:42:59.933859 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 13:42:59.933940 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":80,"threads_started":1}
I20260926 13:43:00.138259 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:00.137763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:44757)
0926 13:43:00.138060 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:00.138139 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
I20260926 13:43:00.789620 16550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:00.789102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:35701)
0926 13:43:00.789428 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:00.789501 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":92,"threads_started":1}
I20260926 13:43:00.935708 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:00.935160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:43541)
0926 13:43:00.935522 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:00.935607 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260926 13:43:01.140095 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:01.139521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:46037)
0926 13:43:01.139886 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:01.139962 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":88,"threads_started":1}
I20260926 13:43:01.791067 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:01.790558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:36005)
0926 13:43:01.790886 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:01.790952 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":118,"threads_started":1}
I20260926 13:43:01.937206 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:01.936708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:37915)
0926 13:43:01.937018 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:01.937091 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":83,"threads_started":1}
I20260926 13:43:02.141601 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:02.141105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:53951)
0926 13:43:02.141427 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:02.141494 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20260926 13:43:02.792702 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:02.792116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:51965)
0926 13:43:02.792423 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:02.792566 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
I20260926 13:43:02.938688 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:02.938170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:34003)
0926 13:43:02.938502 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:02.938559 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":124,"threads_started":1}
I20260926 13:43:03.142989 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:03.142522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:40189)
0926 13:43:03.142828 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:03.142889 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":82,"threads_started":1}
I20260926 13:43:03.794381 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:03.793892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.1:34615)
0926 13:43:03.794205 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:03.794280 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":80,"threads_started":1}
I20260926 13:43:03.940410 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:03.939926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.2:44137)
0926 13:43:03.940239 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:03.940303 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
I20260926 13:43:04.135134 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 16076
I20260926 13:43:04.141294 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 16243
I20260926 13:43:04.144691 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:04.144154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:46133 (local address 127.1.49.3:39915)
0926 13:43:04.144468 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:04.144583 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":94,"threads_started":1}
I20260926 13:43:04.147719 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 16377
2026-09-26T13:43:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19789 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:43:04 dist-test-slave-1wns krb5kdc[16564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:43:04 dist-test-slave-1wns krb5kdc[16564](info): set up 2 sockets
Sep 26 13:43:04 dist-test-slave-1wns krb5kdc[16564](info): commencing operation
krb5kdc: starting...
W20260926 13:43:05.260313 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:43:05 dist-test-slave-1wns krb5kdc[16564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:43:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:43:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:43:05.404105 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:35215
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:35215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:43:05.529073 16577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:43:05.529421 16577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:43:05.529476 16577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:43:05.534214 16577 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 13:43:05.534292 16577 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 13:43:05.534308 16577 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 13:43:05.534323 16577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:43:05.534345 16577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:43:05.534363 16577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:43:05.534379 16577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:43:05.534395 16577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:43:05.540220 16577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:35215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:35215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:43:05.541452 16577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:43:05.542578 16577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:43:05.551991 16577 server_base.cc:1034] running on GCE node
W20260926 13:43:05.551867 16585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:05.551867 16582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:05.551867 16583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:05.552772 16577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:43:05.553970 16577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:43:05.555150 16577 hybrid_clock.cc:648] HybridClock initialized: now 1790430185555137 us; error 29 us; skew 500 ppm
Sep 26 13:43:05 dist-test-slave-1wns krb5kdc[16564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430185, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:43:05.558506 16577 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:43:05.559691 16577 webserver.cc:460] Webserver started at http://127.1.49.62:35829/ using document root <none> and password file <none>
I20260926 13:43:05.560333 16577 fs_manager.cc:359] Metadata directory not provided
I20260926 13:43:05.560402 16577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:43:05.560587 16577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:43:05.562937 16577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "1d9464aefe6143ba95a526c67b41fcbc"
format_stamp: "Formatted at 2026-09-26 13:43:05 on dist-test-slave-1wns"
server_key: "e1d1599f48ed981ed347132813c400b0"
server_key_iv: "5d9eeddf6d91cc3cc1201a95ae0b451a"
server_key_version: "encryptionkey@0"
I20260926 13:43:05.563562 16577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "1d9464aefe6143ba95a526c67b41fcbc"
format_stamp: "Formatted at 2026-09-26 13:43:05 on dist-test-slave-1wns"
server_key: "e1d1599f48ed981ed347132813c400b0"
server_key_iv: "5d9eeddf6d91cc3cc1201a95ae0b451a"
server_key_version: "encryptionkey@0"
I20260926 13:43:05.568001 16577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:43:05.571481 16592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:05.573056 16577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:43:05.573182 16577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "1d9464aefe6143ba95a526c67b41fcbc"
format_stamp: "Formatted at 2026-09-26 13:43:05 on dist-test-slave-1wns"
server_key: "e1d1599f48ed981ed347132813c400b0"
server_key_iv: "5d9eeddf6d91cc3cc1201a95ae0b451a"
server_key_version: "encryptionkey@0"
I20260926 13:43:05.573292 16577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:43:05.587044 16577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:43:05.590883 16577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:43:05.591074 16577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:43:05.599735 16577 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:35215
I20260926 13:43:05.600071 16644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:35215 every 8 connection(s)
I20260926 13:43:05.600901 16577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:43:05.601689 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 16577
I20260926 13:43:05.601830 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:43:05.602195 1220 external_mini_cluster.cc:1428] Setting key cbfb73b562c7b234f96d390239ee2a9a
I20260926 13:43:05.604650 16645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:43:05.611960 16645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Bootstrap starting.
Sep 26 13:43:05 dist-test-slave-1wns krb5kdc[16564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:43:05.614883 16645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Neither blocks nor log segments found. Creating new log.
I20260926 13:43:05.615867 16645 log.cc:824] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Log is configured to *not* fsync() on all Append() calls
I20260926 13:43:05.617929 16645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: No bootstrap required, opened a new log
I20260926 13:43:05.619884 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:05.603950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.1.49.62:35215)
0926 13:43:05.604484 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:05.604509 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:05.604548 (+ 39us) server_negotiation.cc:407] Connection header received
0926 13:43:05.605305 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:05.605323 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:05.605696 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:05.606028 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:43:05.606968 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:05.608331 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:05.609556 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:05.611466 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:05.611610 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:05.614829 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:43:05.614834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:43:05.614850 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:43:05.614885 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:43:05.616752 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:05.617253 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:05.617260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:05.617267 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:05.617391 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:05.617709 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:05.617714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:05.617716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:05.618215 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:43:05.618437 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:05.618745 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 13:43:05.619059 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":158,"threads_started":1}
I20260926 13:43:05.622514 16645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } }
I20260926 13:43:05.622852 16645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:43:05.622917 16645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9464aefe6143ba95a526c67b41fcbc, State: Initialized, Role: FOLLOWER
I20260926 13:43:05.623410 16645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } }
I20260926 13:43:05.623565 16645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:43:05.623651 16645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:43:05.623746 16645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:43:05.624790 16645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } }
I20260926 13:43:05.625169 16645 leader_election.cc:302] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d9464aefe6143ba95a526c67b41fcbc; no voters:
I20260926 13:43:05.625555 16645 leader_election.cc:288] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:43:05.625833 16650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:43:05.626058 16650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 1 LEADER]: Becoming Leader. State: Replica: 1d9464aefe6143ba95a526c67b41fcbc, State: Running, Role: LEADER
I20260926 13:43:05.626493 16650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } }
I20260926 13:43:05.627044 16645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:43:05.628335 16651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d9464aefe6143ba95a526c67b41fcbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } } }
I20260926 13:43:05.628342 16652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d9464aefe6143ba95a526c67b41fcbc. Latest consensus state: current_term: 1 leader_uuid: "1d9464aefe6143ba95a526c67b41fcbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } } }
I20260926 13:43:05.628568 16651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: This master's current role is: LEADER
I20260926 13:43:05.628568 16652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: This master's current role is: LEADER
I20260926 13:43:05.629001 16659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:43:05.632360 16659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:43:05.638902 16659 catalog_manager.cc:1347] Generated new cluster ID: 4dc8bb7e2ae34d789a0dd74ba5aee739
I20260926 13:43:05.638999 16659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:43:05.645475 16659 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:43:05.646804 16659 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:43:05.656935 16659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Generated new TSK 0
I20260926 13:43:05.657769 16659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:43:05.691887 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:35215
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:43:05.809727 16671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:43:05.810016 16671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:43:05.810061 16671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:43:05.814352 16671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:43:05.814437 16671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:43:05.814503 16671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:43:05.820220 16671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:43:05.821388 16671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:43:05.822471 16671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:43:05.832922 16679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:05.832909 16676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:05.832903 16677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:05.833303 16671 server_base.cc:1034] running on GCE node
I20260926 13:43:05.833884 16671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:43:05.834653 16671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:43:05.835911 16671 hybrid_clock.cc:648] HybridClock initialized: now 1790430185835891 us; error 29 us; skew 500 ppm
Sep 26 13:43:05 dist-test-slave-1wns krb5kdc[16564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430185, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:43:05.839195 16671 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:43:05.840301 16671 webserver.cc:460] Webserver started at http://127.1.49.1:44655/ using document root <none> and password file <none>
I20260926 13:43:05.840929 16671 fs_manager.cc:359] Metadata directory not provided
I20260926 13:43:05.840994 16671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:43:05.841180 16671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:43:05.843446 16671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "d243d45394354f9896d873e1e41f6192"
format_stamp: "Formatted at 2026-09-26 13:43:05 on dist-test-slave-1wns"
server_key: "70e96571770c9f8b9fe0a433815b8cfc"
server_key_iv: "775a960c23d6f0f4e1e4737887b2829a"
server_key_version: "encryptionkey@0"
I20260926 13:43:05.844115 16671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "d243d45394354f9896d873e1e41f6192"
format_stamp: "Formatted at 2026-09-26 13:43:05 on dist-test-slave-1wns"
server_key: "70e96571770c9f8b9fe0a433815b8cfc"
server_key_iv: "775a960c23d6f0f4e1e4737887b2829a"
server_key_version: "encryptionkey@0"
I20260926 13:43:05.848457 16671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 13:43:05.851310 16686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:05.852831 16671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:43:05.852949 16671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "d243d45394354f9896d873e1e41f6192"
format_stamp: "Formatted at 2026-09-26 13:43:05 on dist-test-slave-1wns"
server_key: "70e96571770c9f8b9fe0a433815b8cfc"
server_key_iv: "775a960c23d6f0f4e1e4737887b2829a"
server_key_version: "encryptionkey@0"
I20260926 13:43:05.853061 16671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:43:05.867282 16671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:43:05.870654 16671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:43:05.870821 16671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:43:05.871436 16671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:43:05.872344 16671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:43:05.872396 16671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:05.872437 16671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:43:05.872452 16671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:05.883134 16671 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:44575
I20260926 13:43:05.883183 16799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:44575 every 8 connection(s)
I20260926 13:43:05.884219 16671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:43:05.889091 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 16671
I20260926 13:43:05.889199 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:43:05.889489 1220 external_mini_cluster.cc:1428] Setting key 5ac34f5b5d26b5a1b5ca8e19ab71a6d6
Sep 26 13:43:05 dist-test-slave-1wns krb5kdc[16564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430185, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
I20260926 13:43:05.898962 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:05.886265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:32835 (local address 127.1.49.62:35215)
0926 13:43:05.886413 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:05.886417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:05.887447 (+ 1030us) server_negotiation.cc:407] Connection header received
0926 13:43:05.888486 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:05.888490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:05.888546 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:05.888625 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:43:05.890066 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:05.890690 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:05.892056 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:05.892320 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:05.892478 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:05.895113 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:43:05.895117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:43:05.895119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:43:05.895154 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:43:05.896749 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:05.897296 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:05.897300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:05.897301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:05.897385 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:05.897749 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:05.897753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:05.897755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:05.897982 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:43:05.898095 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:05.898649 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 13:43:05.898789 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:43:05.899886 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:05.886549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:32835)
0926 13:43:05.887285 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:05.887331 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:05.888246 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:05.888773 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:05.888780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:05.889212 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:43:05.889785 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:05.889801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:05.890827 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:05.890831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:05.891931 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:05.891941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:05.892511 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:05.892515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:05.892704 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:05.893348 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:43:05.893366 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:43:05.894936 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:43:05.896876 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:05.896883 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:05.896901 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:05.897181 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:05.897479 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:05.897483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:05.897485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:05.897644 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:05.898098 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:43:05.898107 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:43:05.898403 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 13:43:05.898607 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 13:43:05.898959 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":125,"threads_started":1}
I20260926 13:43:05.901152 16800 heartbeater.cc:347] Connected to a master server at 127.1.49.62:35215
I20260926 13:43:05.901456 16800 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
I20260926 13:43:05.902163 16800 heartbeater.cc:511] Master 127.1.49.62:35215 requested a full tablet report, sending...
I20260926 13:43:05.904088 16609 ts_manager.cc:194] Registered new tserver with Master: d243d45394354f9896d873e1e41f6192 (127.1.49.1:44575)
I20260926 13:43:05.905241 16609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:32835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:43:05.919978 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:35215
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:43:06.035459 16805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:43:06.035807 16805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:43:06.035925 16805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:43:06.040223 16805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:43:06.040321 16805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:43:06.040416 16805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:43:06.046168 16805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:43:06.047349 16805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:43:06.048458 16805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:43:06.058965 16811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:06.058947 16810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:06.058947 16813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:06.059000 16805 server_base.cc:1034] running on GCE node
I20260926 13:43:06.059882 16805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:43:06.060735 16805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:43:06.061934 16805 hybrid_clock.cc:648] HybridClock initialized: now 1790430186061917 us; error 37 us; skew 500 ppm
Sep 26 13:43:06 dist-test-slave-1wns krb5kdc[16564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430186, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:43:06.065100 16805 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:43:06.066037 16805 webserver.cc:460] Webserver started at http://127.1.49.2:46663/ using document root <none> and password file <none>
I20260926 13:43:06.066676 16805 fs_manager.cc:359] Metadata directory not provided
I20260926 13:43:06.066742 16805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:43:06.066923 16805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:43:06.069202 16805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "2a0cbf7482a14f849b33b09526a24eb8"
format_stamp: "Formatted at 2026-09-26 13:43:06 on dist-test-slave-1wns"
server_key: "2ca6bc6f72630d057c5ea662d3dbfc05"
server_key_iv: "cc62d2f51aad59b32870982ea7ec184f"
server_key_version: "encryptionkey@0"
I20260926 13:43:06.069886 16805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "2a0cbf7482a14f849b33b09526a24eb8"
format_stamp: "Formatted at 2026-09-26 13:43:06 on dist-test-slave-1wns"
server_key: "2ca6bc6f72630d057c5ea662d3dbfc05"
server_key_iv: "cc62d2f51aad59b32870982ea7ec184f"
server_key_version: "encryptionkey@0"
I20260926 13:43:06.074290 16805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:43:06.077451 16820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:06.079021 16805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:43:06.079211 16805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "2a0cbf7482a14f849b33b09526a24eb8"
format_stamp: "Formatted at 2026-09-26 13:43:06 on dist-test-slave-1wns"
server_key: "2ca6bc6f72630d057c5ea662d3dbfc05"
server_key_iv: "cc62d2f51aad59b32870982ea7ec184f"
server_key_version: "encryptionkey@0"
I20260926 13:43:06.079360 16805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:43:06.090628 16805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:43:06.094302 16805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:43:06.094553 16805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:43:06.095232 16805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:43:06.096237 16805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:43:06.096314 16805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:06.096395 16805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:43:06.096452 16805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:06.105973 16805 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:46021
I20260926 13:43:06.106010 16933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:46021 every 8 connection(s)
I20260926 13:43:06.107036 16805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:43:06.107225 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 16805
I20260926 13:43:06.107342 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:43:06.107719 1220 external_mini_cluster.cc:1428] Setting key 068c96455849272f56748c48f9f1d62f
Sep 26 13:43:06 dist-test-slave-1wns krb5kdc[16564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430186, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
I20260926 13:43:06.124409 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.109148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:45295 (local address 127.1.49.62:35215)
0926 13:43:06.109297 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:06.109302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:06.110747 (+ 1445us) server_negotiation.cc:407] Connection header received
0926 13:43:06.111940 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:06.111943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:06.112011 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:06.112093 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:43:06.113376 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.114262 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.115646 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.116118 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.116261 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.119542 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:43:06.119546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:43:06.119548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:43:06.119578 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:43:06.121489 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:06.122202 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:06.122208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:06.122214 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:06.122316 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:06.122757 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:06.122760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:06.122761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:06.122993 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:43:06.123090 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:06.123894 (+ 804us) server_negotiation.cc:299] Negotiation successful
0926 13:43:06.124150 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:43:06.125092 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.109741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:45295)
0926 13:43:06.110557 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:06.110628 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:06.111685 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:06.112300 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:06.112307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:06.112710 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:43:06.113217 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:06.113230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.114428 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.114431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:06.115500 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:06.115512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.116319 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.116323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:06.116600 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.117386 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:43:06.117407 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:43:06.119355 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:43:06.121740 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:06.121746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:06.121762 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:06.122084 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:06.122424 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:06.122429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:06.122432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:06.122640 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:06.123116 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:43:06.123126 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:43:06.123499 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 13:43:06.123755 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 13:43:06.124123 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":125,"threads_started":1}
I20260926 13:43:06.126660 16934 heartbeater.cc:347] Connected to a master server at 127.1.49.62:35215
I20260926 13:43:06.126952 16934 heartbeater.cc:464] Registering TS with master...
I20260926 13:43:06.127591 16934 heartbeater.cc:511] Master 127.1.49.62:35215 requested a full tablet report, sending...
I20260926 13:43:06.128876 16609 ts_manager.cc:194] Registered new tserver with Master: 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021)
I20260926 13:43:06.129568 16609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:45295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:43:06.143627 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:35215
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:43:06.264011 16939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:43:06.264313 16939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:43:06.264360 16939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:43:06.268771 16939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:43:06.268846 16939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:43:06.268937 16939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:43:06.274585 16939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:43:06.275727 16939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:43:06.276822 16939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:43:06.287142 16945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:06.287223 16944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:06.287307 16939 server_base.cc:1034] running on GCE node
W20260926 13:43:06.287520 16947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:06.288129 16939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:43:06.288866 16939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:43:06.290126 16939 hybrid_clock.cc:648] HybridClock initialized: now 1790430186290098 us; error 35 us; skew 500 ppm
Sep 26 13:43:06 dist-test-slave-1wns krb5kdc[16564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430186, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:43:06.293670 16939 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:43:06.294692 16939 webserver.cc:460] Webserver started at http://127.1.49.3:35079/ using document root <none> and password file <none>
I20260926 13:43:06.295298 16939 fs_manager.cc:359] Metadata directory not provided
I20260926 13:43:06.295395 16939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:43:06.295629 16939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:43:06.297873 16939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "29ec3485bf1a4b61bc9bdb727aa67972"
format_stamp: "Formatted at 2026-09-26 13:43:06 on dist-test-slave-1wns"
server_key: "53a9da82a9942033ed3948db6aeae1a4"
server_key_iv: "74c4d3e149fbab79ae4996f3ad223de7"
server_key_version: "encryptionkey@0"
I20260926 13:43:06.298470 16939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "29ec3485bf1a4b61bc9bdb727aa67972"
format_stamp: "Formatted at 2026-09-26 13:43:06 on dist-test-slave-1wns"
server_key: "53a9da82a9942033ed3948db6aeae1a4"
server_key_iv: "74c4d3e149fbab79ae4996f3ad223de7"
server_key_version: "encryptionkey@0"
I20260926 13:43:06.302685 16939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 13:43:06.305696 16954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:06.307318 16939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:43:06.307504 16939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "29ec3485bf1a4b61bc9bdb727aa67972"
format_stamp: "Formatted at 2026-09-26 13:43:06 on dist-test-slave-1wns"
server_key: "53a9da82a9942033ed3948db6aeae1a4"
server_key_iv: "74c4d3e149fbab79ae4996f3ad223de7"
server_key_version: "encryptionkey@0"
I20260926 13:43:06.307667 16939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:43:06.346812 16939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:43:06.350293 16939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:43:06.350504 16939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:43:06.351178 16939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:43:06.352046 16939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:43:06.352118 16939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:06.352190 16939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:43:06.352245 16939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:06.361790 16939 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:44803
I20260926 13:43:06.361807 17067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:44803 every 8 connection(s)
I20260926 13:43:06.362742 16939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:43:06.371611 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 16939
I20260926 13:43:06.371783 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:43:06.372133 1220 external_mini_cluster.cc:1428] Setting key 7983f0a883be0a19c71362f140c0cb8e
Sep 26 13:43:06 dist-test-slave-1wns krb5kdc[16564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430186, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:43:06.379318 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.364785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:37093 (local address 127.1.49.62:35215)
0926 13:43:06.364943 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:06.364948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:06.365907 (+ 959us) server_negotiation.cc:407] Connection header received
0926 13:43:06.367424 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:06.367427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:06.367480 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:06.367560 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:43:06.369699 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.370307 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.371855 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.372115 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.372247 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.375578 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:43:06.375582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:43:06.375584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:43:06.375640 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:43:06.377126 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:06.377666 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:06.377668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:06.377670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:06.377746 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:06.378215 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:06.378217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:06.378218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:06.378422 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:43:06.378499 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:06.379079 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 13:43:06.379194 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 13:43:06.380383 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.365216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:37093)
0926 13:43:06.365740 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:06.365788 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:06.367189 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:06.367778 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:06.367786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:06.368749 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:43:06.369527 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:06.369544 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.370456 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.370459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:06.371668 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:06.371679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.372296 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.372301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:06.372560 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.373391 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:43:06.373415 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:43:06.375337 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:43:06.377262 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:06.377267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:06.377283 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:06.377582 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:06.377867 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:06.377870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:06.377872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:06.378099 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:06.378537 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:43:06.378543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:43:06.378831 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 13:43:06.379027 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 13:43:06.379331 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":119,"threads_started":1}
I20260926 13:43:06.381673 17068 heartbeater.cc:347] Connected to a master server at 127.1.49.62:35215
I20260926 13:43:06.381955 17068 heartbeater.cc:464] Registering TS with master...
I20260926 13:43:06.382522 17068 heartbeater.cc:511] Master 127.1.49.62:35215 requested a full tablet report, sending...
I20260926 13:43:06.383916 16609 ts_manager.cc:194] Registered new tserver with Master: 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803)
I20260926 13:43:06.384548 16609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:37093
I20260926 13:43:06.386683 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:43:06.397445 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.388864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.1.49.62:35215)
0926 13:43:06.389039 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:06.389044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:06.389380 (+ 336us) server_negotiation.cc:407] Connection header received
0926 13:43:06.389479 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:06.389482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:06.389533 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:06.389618 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:43:06.390516 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.391124 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.392129 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.392368 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.392483 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.394061 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:43:06.394065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:43:06.394068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:43:06.394108 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:43:06.395544 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:06.396055 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:06.396058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:06.396059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:06.396166 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:06.396494 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:06.396496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:06.396497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:06.396774 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:43:06.396986 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:06.397128 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 13:43:06.397235 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 13:43:06.401024 16609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:43:06.404315 16609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:43:06.420222 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.414156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.1.49.3:44803)
0926 13:43:06.415399 (+ 1243us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:06.415412 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:06.415437 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:43:06.415524 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:06.415529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:06.415750 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:06.416228 (+ 478us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:43:06.416406 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.417230 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.418768 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.419457 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.419565 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.419712 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:06.419926 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 13:43:06.420029 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1132,"thread_start_us":137,"threads_started":1}
I20260926 13:43:06.420213 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.414052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:44803 (local address 127.0.0.1:35238)
0926 13:43:06.414504 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:06.414556 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:06.414691 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:06.415956 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:06.415962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:06.416000 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:43:06.416243 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:06.416252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.417369 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.417375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:06.418642 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:06.418651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.419572 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.419576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:06.419690 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.419732 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 13:43:06.419887 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 13:43:06.420026 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":76,"spinlock_wait_cycles":9856,"thread_start_us":89,"threads_started":1}
I20260926 13:43:06.420925 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.415001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.1.49.2:46021)
0926 13:43:06.415467 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:06.415478 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:06.415495 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:43:06.415558 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:06.415562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:06.415750 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:06.415934 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:43:06.416391 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.417675 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.419212 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.420084 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.420195 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.420352 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:06.420666 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 13:43:06.420771 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":103,"threads_started":1}
I20260926 13:43:06.421018 17082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.414853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:44575 (local address 127.0.0.1:42834)
0926 13:43:06.415638 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:06.415650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:06.415772 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:06.416376 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:06.416379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:06.416397 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:43:06.416479 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:06.416484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.418185 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.418188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:06.419329 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:06.419337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.420622 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.420627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:06.420715 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.420737 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:43:06.420809 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 13:43:06.420874 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260926 13:43:06.421519 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.415274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.1.49.1:44575)
0926 13:43:06.415516 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:06.415525 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:06.415749 (+ 224us) server_negotiation.cc:407] Connection header received
0926 13:43:06.415941 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:06.415947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:06.416180 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:06.416341 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:43:06.416589 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.418048 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.419454 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.420275 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.420386 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.420540 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:06.421226 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 13:43:06.421351 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":69,"threads_started":1}
I20260926 13:43:06.423416 17002 tablet_service.cc:1467] Processing CreateTablet for tablet aa8b95e38fee43f7a037dd4f6e8ff55f (DEFAULT_TABLE table=test-table [id=368abef3d72346d6949ddcdf5d15e244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:43:06.424064 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.414092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:46021 (local address 127.0.0.1:41716)
0926 13:43:06.414960 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:06.414976 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:06.415058 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:06.415954 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:06.415960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:06.415994 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:43:06.416209 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:06.416217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.417831 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.417834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:06.419074 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:06.419085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.420427 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.420431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:06.420526 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.420547 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:43:06.420628 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 13:43:06.423926 (+ 3298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":61,"spinlock_wait_cycles":3456,"thread_start_us":131,"threads_started":1}
I20260926 13:43:06.423770 16734 tablet_service.cc:1467] Processing CreateTablet for tablet aa8b95e38fee43f7a037dd4f6e8ff55f (DEFAULT_TABLE table=test-table [id=368abef3d72346d6949ddcdf5d15e244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:43:06.424640 17002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8b95e38fee43f7a037dd4f6e8ff55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:43:06.424829 16734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8b95e38fee43f7a037dd4f6e8ff55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:43:06.427107 16868 tablet_service.cc:1467] Processing CreateTablet for tablet aa8b95e38fee43f7a037dd4f6e8ff55f (DEFAULT_TABLE table=test-table [id=368abef3d72346d6949ddcdf5d15e244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:43:06.428272 16868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8b95e38fee43f7a037dd4f6e8ff55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:43:06.431890 17085 tablet_bootstrap.cc:492] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: Bootstrap starting.
I20260926 13:43:06.432982 17086 tablet_bootstrap.cc:492] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972: Bootstrap starting.
I20260926 13:43:06.434401 17085 tablet_bootstrap.cc:654] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: Neither blocks nor log segments found. Creating new log.
I20260926 13:43:06.435211 17085 log.cc:824] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: Log is configured to *not* fsync() on all Append() calls
I20260926 13:43:06.435667 17086 tablet_bootstrap.cc:654] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972: Neither blocks nor log segments found. Creating new log.
I20260926 13:43:06.436519 17086 log.cc:824] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972: Log is configured to *not* fsync() on all Append() calls
I20260926 13:43:06.436643 17088 tablet_bootstrap.cc:492] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8: Bootstrap starting.
I20260926 13:43:06.437721 17085 tablet_bootstrap.cc:492] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: No bootstrap required, opened a new log
I20260926 13:43:06.437989 17085 ts_tablet_manager.cc:1397] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:43:06.438807 17086 tablet_bootstrap.cc:492] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972: No bootstrap required, opened a new log
I20260926 13:43:06.439033 17086 ts_tablet_manager.cc:1397] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260926 13:43:06.439580 17088 tablet_bootstrap.cc:654] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8: Neither blocks nor log segments found. Creating new log.
I20260926 13:43:06.440665 17088 log.cc:824] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:43:06.442621 17086 raft_consensus.cc:348] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:06.442934 17086 raft_consensus.cc:374] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:43:06.442996 17086 raft_consensus.cc:729] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ec3485bf1a4b61bc9bdb727aa67972, State: Initialized, Role: FOLLOWER
I20260926 13:43:06.443091 17088 tablet_bootstrap.cc:492] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8: No bootstrap required, opened a new log
I20260926 13:43:06.443320 17088 ts_tablet_manager.cc:1397] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260926 13:43:06.443219 17085 raft_consensus.cc:348] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:06.443493 17085 raft_consensus.cc:374] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:43:06.443571 17085 raft_consensus.cc:729] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d243d45394354f9896d873e1e41f6192, State: Initialized, Role: FOLLOWER
I20260926 13:43:06.443536 17086 consensus_queue.cc:260] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:06.444095 17085 consensus_queue.cc:260] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:06.444463 17068 heartbeater.cc:503] Master 127.1.49.62:35215 was elected leader, sending a full tablet report...
I20260926 13:43:06.444686 17086 ts_tablet_manager.cc:1428] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260926 13:43:06.444841 16800 heartbeater.cc:503] Master 127.1.49.62:35215 was elected leader, sending a full tablet report...
I20260926 13:43:06.445214 17085 ts_tablet_manager.cc:1428] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 13:43:06.449038 17088 raft_consensus.cc:348] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:06.449438 17088 raft_consensus.cc:374] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:43:06.449517 17088 raft_consensus.cc:729] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0cbf7482a14f849b33b09526a24eb8, State: Initialized, Role: FOLLOWER
I20260926 13:43:06.449980 17088 consensus_queue.cc:260] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:06.450707 16934 heartbeater.cc:503] Master 127.1.49.62:35215 was elected leader, sending a full tablet report...
I20260926 13:43:06.451169 17088 ts_tablet_manager.cc:1428] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260926 13:43:06.539192 17092 raft_consensus.cc:482] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:43:06.539463 17092 raft_consensus.cc:504] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:06.540802 17092 leader_election.cc:288] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803), 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021)
I20260926 13:43:06.546559 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.541274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:36557 (local address 127.1.49.3:44803)
0926 13:43:06.541413 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:06.541418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:06.541703 (+ 285us) server_negotiation.cc:407] Connection header received
0926 13:43:06.541835 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:06.541838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:06.541895 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:06.541977 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:43:06.542375 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.543090 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.545030 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.545724 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.545847 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.545917 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:06.546317 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 13:43:06.546406 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 13:43:06.546559 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.541274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:35891 (local address 127.1.49.2:46021)
0926 13:43:06.541445 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:06.541449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:06.541845 (+ 396us) server_negotiation.cc:407] Connection header received
0926 13:43:06.541956 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:06.541959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:06.542044 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:06.542125 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:43:06.542483 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.543190 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.545097 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.545725 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.545855 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.545917 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:06.546317 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 13:43:06.546406 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:43:06.546640 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.541148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:44803 (local address 127.1.49.1:36557)
0926 13:43:06.541588 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:06.541600 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:06.541720 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:06.542060 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:06.542065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:06.542086 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:43:06.542260 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:06.542267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.543221 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.543224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:06.544862 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:06.544874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.545907 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.545911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:06.546116 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.546165 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 13:43:06.546262 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 13:43:06.546406 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":252,"thread_start_us":129,"threads_started":1}
I20260926 13:43:06.546640 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.541148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:46021 (local address 127.1.49.1:35891)
0926 13:43:06.541715 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:06.541731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:06.541846 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:06.542163 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:06.542165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:06.542179 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:43:06.542301 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:06.542327 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.543305 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.543307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:06.544862 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:06.544874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.545907 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.545911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:06.546112 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.546162 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 13:43:06.546269 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 13:43:06.546407 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":236,"thread_start_us":91,"threads_started":1}
I20260926 13:43:06.547421 17022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8b95e38fee43f7a037dd4f6e8ff55f" candidate_uuid: "d243d45394354f9896d873e1e41f6192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" is_pre_election: true
I20260926 13:43:06.547477 16888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8b95e38fee43f7a037dd4f6e8ff55f" candidate_uuid: "d243d45394354f9896d873e1e41f6192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a0cbf7482a14f849b33b09526a24eb8" is_pre_election: true
I20260926 13:43:06.547842 16888 raft_consensus.cc:2454] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d243d45394354f9896d873e1e41f6192 in term 0.
I20260926 13:43:06.547842 17022 raft_consensus.cc:2454] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d243d45394354f9896d873e1e41f6192 in term 0.
I20260926 13:43:06.548471 16690 leader_election.cc:302] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29ec3485bf1a4b61bc9bdb727aa67972, 2a0cbf7482a14f849b33b09526a24eb8, d243d45394354f9896d873e1e41f6192; no voters:
I20260926 13:43:06.548834 17092 raft_consensus.cc:2781] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:43:06.548949 17092 raft_consensus.cc:482] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:43:06.549023 17092 raft_consensus.cc:3037] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:43:06.550125 17092 raft_consensus.cc:504] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:06.550513 17092 leader_election.cc:288] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 1 election: Requested vote from peers 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803), 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021)
I20260926 13:43:06.550972 16888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8b95e38fee43f7a037dd4f6e8ff55f" candidate_uuid: "d243d45394354f9896d873e1e41f6192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a0cbf7482a14f849b33b09526a24eb8"
I20260926 13:43:06.550972 17022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8b95e38fee43f7a037dd4f6e8ff55f" candidate_uuid: "d243d45394354f9896d873e1e41f6192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ec3485bf1a4b61bc9bdb727aa67972"
I20260926 13:43:06.551126 17022 raft_consensus.cc:3037] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:43:06.551126 16888 raft_consensus.cc:3037] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:43:06.552564 17022 raft_consensus.cc:2454] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d243d45394354f9896d873e1e41f6192 in term 1.
I20260926 13:43:06.552564 16888 raft_consensus.cc:2454] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d243d45394354f9896d873e1e41f6192 in term 1.
I20260926 13:43:06.552958 16688 leader_election.cc:302] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29ec3485bf1a4b61bc9bdb727aa67972, 2a0cbf7482a14f849b33b09526a24eb8, d243d45394354f9896d873e1e41f6192; no voters:
I20260926 13:43:06.553131 17092 raft_consensus.cc:2781] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:43:06.553387 17092 raft_consensus.cc:686] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 1 LEADER]: Becoming Leader. State: Replica: d243d45394354f9896d873e1e41f6192, State: Running, Role: LEADER
I20260926 13:43:06.553791 17092 consensus_queue.cc:237] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:06.557307 16608 catalog_manager.cc:5495] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 reported cstate change: term changed from 0 to 1, leader changed from <none> to d243d45394354f9896d873e1e41f6192 (127.1.49.1). New cstate: current_term: 1 leader_uuid: "d243d45394354f9896d873e1e41f6192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } health_report { overall_health: HEALTHY } } }
I20260926 13:43:06.582934 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.578835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.1.49.1:44575)
0926 13:43:06.579008 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:06.579012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:06.579071 (+ 59us) server_negotiation.cc:407] Connection header received
0926 13:43:06.579154 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:06.579156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:06.579232 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:06.579360 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:43:06.579645 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.580278 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.581257 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:06.581503 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:06.581625 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:06.581973 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:43:06.582479 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:43:06.582620 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:06.582692 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 13:43:06.582759 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 13:43:06.589732 16888 raft_consensus.cc:1261] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8 [term 1 FOLLOWER]: Refusing update from remote peer d243d45394354f9896d873e1e41f6192: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:43:06.589732 17022 raft_consensus.cc:1261] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972 [term 1 FOLLOWER]: Refusing update from remote peer d243d45394354f9896d873e1e41f6192: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:43:06.590617 17096 consensus_queue.cc:1035] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:43:06.590838 17092 consensus_queue.cc:1035] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:43:06.598387 17099 mvcc.cc:205] Tried to move back new op lower bound from 7333602044263022592 to 7333602044127326208. Current Snapshot: MvccSnapshot[applied={T|T < 7333602044263022592}]
I20260926 13:43:06.601253 17100 mvcc.cc:205] Tried to move back new op lower bound from 7333602044263022592 to 7333602044127326208. Current Snapshot: MvccSnapshot[applied={T|T < 7333602044263022592}]
I20260926 13:43:06.604245 17101 mvcc.cc:205] Tried to move back new op lower bound from 7333602044263022592 to 7333602044127326208. Current Snapshot: MvccSnapshot[applied={T|T < 7333602044263022592 or (T in {7333602044263022592})}]
I20260926 13:43:06.608270 16608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:43:06.608460 16608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:43:06.611474 16608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Sending DeleteTablet for 3 replicas of tablet aa8b95e38fee43f7a037dd4f6e8ff55f
I20260926 13:43:06.612537 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 16671
I20260926 13:43:06.614311 16868 tablet_service.cc:1514] Processing DeleteTablet for tablet aa8b95e38fee43f7a037dd4f6e8ff55f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:43:06 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:41716
I20260926 13:43:06.614931 17108 tablet_replica.cc:330] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8: stopping tablet replica
I20260926 13:43:06.615272 17002 tablet_service.cc:1514] Processing DeleteTablet for tablet aa8b95e38fee43f7a037dd4f6e8ff55f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:43:06 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:35238
I20260926 13:43:06.615342 17108 raft_consensus.cc:2229] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:43:06.615808 17108 raft_consensus.cc:2258] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:43:06.615892 17109 tablet_replica.cc:330] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972: stopping tablet replica
I20260926 13:43:06.616410 17109 raft_consensus.cc:2229] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:43:06.616747 17109 raft_consensus.cc:2258] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:43:06.618374 17109 ts_tablet_manager.cc:1905] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:43:06.621448 17109 ts_tablet_manager.cc:1918] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:43:06.621567 17109 log.cc:1196] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/aa8b95e38fee43f7a037dd4f6e8ff55f
I20260926 13:43:06.621672 17108 ts_tablet_manager.cc:1905] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:43:06.621937 17109 ts_tablet_manager.cc:1939] T aa8b95e38fee43f7a037dd4f6e8ff55f P 29ec3485bf1a4b61bc9bdb727aa67972: Deleting consensus metadata
W20260926 13:43:06.622453 16596 connection.cc:535] client connection to 127.1.49.1:44575 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 13:43:06.622606 16596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:43:06.623407 16596 catalog_manager.cc:4841] TS 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803): tablet aa8b95e38fee43f7a037dd4f6e8ff55f (table test-table [id=368abef3d72346d6949ddcdf5d15e244]) successfully deleted
I20260926 13:43:06.623986 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 16805
I20260926 13:43:06.626868 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.626491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:44575 (local address 127.0.0.1:42854)
0926 13:43:06.626730 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:06.626793 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260926 13:43:06.627091 16596 catalog_manager.cc:4584] TS d243d45394354f9896d873e1e41f6192 (127.1.49.1:44575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa8b95e38fee43f7a037dd4f6e8ff55f: Network error: Client connection negotiation failed: client connection to 127.1.49.1:44575: connect: Connection refused (error 111)
I20260926 13:43:06.631738 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 16939
I20260926 13:43:06.632581 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:06.631981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:46021 (local address 127.0.0.1:41724)
0926 13:43:06.632383 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:06.632449 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.2:46021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304}
I20260926 13:43:06.637435 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 16577
I20260926 13:43:26.644723 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:35215
--webserver_interface=127.1.49.62
--webserver_port=35829
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:35215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:43:26.776226 17111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:43:26.776563 17111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:43:26.776609 17111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:43:26.781037 17111 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 13:43:26.781117 17111 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 13:43:26.781136 17111 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 13:43:26.781149 17111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:43:26.781170 17111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:43:26.781188 17111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:43:26.781206 17111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:43:26.781265 17111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:43:26.787252 17111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:35215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:35215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:43:26.788667 17111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:43:26.789846 17111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:43:26.804683 17117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:26.804637 17120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:26.804620 17118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:26.804725 17111 server_base.cc:1034] running on GCE node
I20260926 13:43:26.805848 17111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:43:26.807595 17111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:43:26.808832 17111 hybrid_clock.cc:648] HybridClock initialized: now 1790430206808817 us; error 31 us; skew 500 ppm
Sep 26 13:43:26 dist-test-slave-1wns krb5kdc[16564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430206, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:43:26.812417 17111 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:43:26.813560 17111 webserver.cc:460] Webserver started at http://127.1.49.62:35829/ using document root <none> and password file <none>
I20260926 13:43:26.814221 17111 fs_manager.cc:359] Metadata directory not provided
I20260926 13:43:26.814301 17111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:43:26.819015 17111 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:43:26.821946 17127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:26.823951 17111 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:43:26.824115 17111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "1d9464aefe6143ba95a526c67b41fcbc"
format_stamp: "Formatted at 2026-09-26 13:43:05 on dist-test-slave-1wns"
server_key: "e1d1599f48ed981ed347132813c400b0"
server_key_iv: "5d9eeddf6d91cc3cc1201a95ae0b451a"
server_key_version: "encryptionkey@0"
I20260926 13:43:26.824970 17111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:43:26.849834 17111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:43:26.853710 17111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:43:26.853899 17111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:43:26.862336 17111 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:35215
I20260926 13:43:26.862706 17179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:35215 every 8 connection(s)
I20260926 13:43:26.863826 17111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:43:26.869459 17180 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:43:26.871585 17180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Bootstrap starting.
I20260926 13:43:26.873224 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 17111
I20260926 13:43:26.873718 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:44575
--local_ip_for_outbound_sockets=127.1.49.1
--tserver_master_addrs=127.1.49.62:35215
--webserver_port=44655
--webserver_interface=127.1.49.1
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:43:26.891397 17180 log.cc:824] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Log is configured to *not* fsync() on all Append() calls
I20260926 13:43:26.901825 17180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:43:26.902285 17180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Bootstrap complete.
I20260926 13:43:26.907069 17180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } }
I20260926 13:43:26.907583 17180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9464aefe6143ba95a526c67b41fcbc, State: Initialized, Role: FOLLOWER
I20260926 13:43:26.908319 17180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } }
I20260926 13:43:26.908501 17180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:43:26.908618 17180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:43:26.908759 17180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:43:26.911054 17180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } }
I20260926 13:43:26.911492 17180 leader_election.cc:302] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d9464aefe6143ba95a526c67b41fcbc; no voters:
I20260926 13:43:26.911926 17180 leader_election.cc:288] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:43:26.912027 17184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:43:26.912302 17184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 2 LEADER]: Becoming Leader. State: Replica: 1d9464aefe6143ba95a526c67b41fcbc, State: Running, Role: LEADER
I20260926 13:43:26.912655 17184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } }
I20260926 13:43:26.913462 17180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:43:26.913758 17184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d9464aefe6143ba95a526c67b41fcbc. Latest consensus state: current_term: 2 leader_uuid: "1d9464aefe6143ba95a526c67b41fcbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } } }
I20260926 13:43:26.913883 17184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: This master's current role is: LEADER
I20260926 13:43:26.914285 17185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1d9464aefe6143ba95a526c67b41fcbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } } }
I20260926 13:43:26.914515 17185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: This master's current role is: LEADER
I20260926 13:43:26.914306 17188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:43:26.920970 17188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:43:26.921793 17188 catalog_manager.cc:1259] Loaded cluster ID: 4dc8bb7e2ae34d789a0dd74ba5aee739
I20260926 13:43:26.921867 17188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:43:26.924782 17188 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:43:26.925614 17188 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Loaded TSK: 0
I20260926 13:43:26.941004 17188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Generated new TSK 1
I20260926 13:43:26.941318 17188 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:43:27.007597 17182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:43:27.007956 17182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:43:27.008054 17182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:43:27.012583 17182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:43:27.012669 17182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:43:27.012777 17182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:43:27.019304 17182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:44575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=44655
--tserver_master_addrs=127.1.49.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:43:27.020541 17182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:43:27.021628 17182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:43:27.032421 17208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:27.032758 17207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:27.032454 17210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:27.033339 17182 server_base.cc:1034] running on GCE node
I20260926 13:43:27.033939 17182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:43:27.034891 17182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:43:27.036110 17182 hybrid_clock.cc:648] HybridClock initialized: now 1790430207036069 us; error 55 us; skew 500 ppm
Sep 26 13:43:27 dist-test-slave-1wns krb5kdc[16564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430207, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:43:27.040548 17182 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:43:27.042124 17182 webserver.cc:460] Webserver started at http://127.1.49.1:44655/ using document root <none> and password file <none>
I20260926 13:43:27.042991 17182 fs_manager.cc:359] Metadata directory not provided
I20260926 13:43:27.043119 17182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:43:27.049546 17182 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:43:27.056962 17217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:27.059005 17182 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260926 13:43:27.059216 17182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "d243d45394354f9896d873e1e41f6192"
format_stamp: "Formatted at 2026-09-26 13:43:05 on dist-test-slave-1wns"
server_key: "70e96571770c9f8b9fe0a433815b8cfc"
server_key_iv: "775a960c23d6f0f4e1e4737887b2829a"
server_key_version: "encryptionkey@0"
I20260926 13:43:27.059968 17182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:43:27.094242 17182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:43:27.098239 17182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:43:27.098487 17182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:43:27.099247 17182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:43:27.100601 17224 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:43:27.108357 17182 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:43:27.108466 17182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260926 13:43:27.108546 17182 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:43:27.110592 17182 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:43:27.110671 17182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 13:43:27.110951 17224 tablet_bootstrap.cc:492] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: Bootstrap starting.
I20260926 13:43:27.123145 17331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:44575 every 8 connection(s)
I20260926 13:43:27.123147 17182 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:44575
I20260926 13:43:27.124699 17182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:43:27.132797 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 17182
I20260926 13:43:27.133287 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:46021
--local_ip_for_outbound_sockets=127.1.49.2
--tserver_master_addrs=127.1.49.62:35215
--webserver_port=46663
--webserver_interface=127.1.49.2
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:43:27.133867 17224 log.cc:824] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: Log is configured to *not* fsync() on all Append() calls
Sep 26 13:43:27 dist-test-slave-1wns krb5kdc[16564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430207, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:43:27.142753 17224 tablet_bootstrap.cc:492] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:43:27.143276 17224 tablet_bootstrap.cc:492] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: Bootstrap complete.
I20260926 13:43:27.144655 17224 ts_tablet_manager.cc:1397] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.007s
I20260926 13:43:27.146494 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.127911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:42187)
0926 13:43:27.128463 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:27.128510 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:27.129464 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:27.130744 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:27.130752 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:27.131248 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:43:27.131769 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.131785 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.133674 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.133679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.135072 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.135084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.135989 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.135993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.136208 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.136945 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:43:27.136972 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:43:27.139005 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:43:27.142204 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:27.142211 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:27.142224 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:27.142598 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:27.143141 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:27.143147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:27.143152 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:27.143328 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:27.144175 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:43:27.144185 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:43:27.144565 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 13:43:27.144866 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 13:43:27.145323 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":150,"threads_started":1}
I20260926 13:43:27.147209 17224 raft_consensus.cc:348] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:27.147686 17224 raft_consensus.cc:729] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d243d45394354f9896d873e1e41f6192, State: Initialized, Role: FOLLOWER
I20260926 13:43:27.148203 17224 consensus_queue.cc:260] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:27.148614 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.127754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:42187 (local address 127.1.49.62:35215)
0926 13:43:27.128223 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:27.128245 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:27.128649 (+ 404us) server_negotiation.cc:407] Connection header received
0926 13:43:27.129957 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:27.129968 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:27.130327 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:27.130688 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:43:27.132026 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.133489 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.135244 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.135768 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.136031 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.139265 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:43:27.139275 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:43:27.139298 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:43:27.139345 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:43:27.142016 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:27.142740 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:27.142749 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:27.142758 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:27.143016 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:27.143453 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:27.143457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:27.143459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:27.144025 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:43:27.144167 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:27.147232 (+ 3065us) server_negotiation.cc:299] Negotiation successful
0926 13:43:27.147576 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":163,"threads_started":1}
I20260926 13:43:27.149267 17224 ts_tablet_manager.cc:1428] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 13:43:27.150910 17332 heartbeater.cc:347] Connected to a master server at 127.1.49.62:35215
I20260926 13:43:27.151170 17332 heartbeater.cc:464] Registering TS with master...
I20260926 13:43:27.151762 17332 heartbeater.cc:511] Master 127.1.49.62:35215 requested a full tablet report, sending...
I20260926 13:43:27.154852 17144 ts_manager.cc:194] Registered new tserver with Master: d243d45394354f9896d873e1e41f6192 (127.1.49.1:44575)
I20260926 13:43:27.159364 17144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:42187
I20260926 13:43:27.162516 17332 heartbeater.cc:503] Master 127.1.49.62:35215 was elected leader, sending a full tablet report...
Sep 26 13:43:27 dist-test-slave-1wns krb5kdc[16564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430206, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for kudu/127.1.49.1@KRBTEST.COM
I20260926 13:43:27.172969 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.158624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:44575 (local address 127.0.0.1:43740)
0926 13:43:27.159125 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:27.159177 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:27.159322 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:27.160623 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:27.160629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:27.163182 (+ 2553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:43:27.163343 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.163353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.164796 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.164803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.165453 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.165471 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.166151 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.166171 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.166300 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.166915 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:43:27.166929 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:43:27.168781 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:43:27.170848 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:27.170858 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:27.170864 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:27.171139 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:27.171465 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:27.171468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:27.171470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:27.171620 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:27.172232 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:43:27.172239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:43:27.172374 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 13:43:27.172520 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 13:43:27.172771 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":143,"threads_started":1}
I20260926 13:43:27.172969 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.159608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43740 (local address 127.1.49.1:44575)
0926 13:43:27.160037 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:27.160050 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:27.160078 (+ 28us) server_negotiation.cc:407] Connection header received
0926 13:43:27.160164 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:27.160170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:27.160438 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:27.160606 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:43:27.163500 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.164679 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.165594 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.166020 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.166156 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.168962 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:43:27.168972 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:43:27.168983 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:43:27.169009 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:43:27.170703 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:27.171272 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:27.171277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:27.171278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:27.171361 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:27.171721 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:27.171725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:27.171727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:27.172108 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:43:27.172261 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:27.172616 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 13:43:27.172813 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":124,"threads_started":1}
I20260926 13:43:27.174420 17266 tablet_service.cc:1514] Processing DeleteTablet for tablet aa8b95e38fee43f7a037dd4f6e8ff55f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:43:06 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:43740
I20260926 13:43:27.175432 17341 tablet_replica.cc:330] stopping tablet replica
I20260926 13:43:27.175832 17341 raft_consensus.cc:2229] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:43:27.176117 17341 raft_consensus.cc:2258] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:43:27.177310 17341 ts_tablet_manager.cc:1905] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:43:27.182199 17341 ts_tablet_manager.cc:1918] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:43:27.182319 17341 log.cc:1196] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal/wals/aa8b95e38fee43f7a037dd4f6e8ff55f
I20260926 13:43:27.182603 17341 ts_tablet_manager.cc:1939] T aa8b95e38fee43f7a037dd4f6e8ff55f P d243d45394354f9896d873e1e41f6192: Deleting consensus metadata
I20260926 13:43:27.184975 17131 catalog_manager.cc:4841] TS d243d45394354f9896d873e1e41f6192 (127.1.49.1:44575): tablet aa8b95e38fee43f7a037dd4f6e8ff55f (table test-table [id=368abef3d72346d6949ddcdf5d15e244]) successfully deleted
W20260926 13:43:27.265525 17337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:43:27.265987 17337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:43:27.266073 17337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:43:27.272395 17337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:43:27.272523 17337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:43:27.272641 17337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:43:27.278934 17337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:46021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=46663
--tserver_master_addrs=127.1.49.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:43:27.280308 17337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:43:27.281561 17337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:43:27.293053 17349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:27.293098 17346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:27.293085 17347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:27.293252 17337 server_base.cc:1034] running on GCE node
I20260926 13:43:27.293834 17337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:43:27.294580 17337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:43:27.295836 17337 hybrid_clock.cc:648] HybridClock initialized: now 1790430207295810 us; error 46 us; skew 500 ppm
Sep 26 13:43:27 dist-test-slave-1wns krb5kdc[16564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430207, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:43:27.299925 17337 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:43:27.301157 17337 webserver.cc:460] Webserver started at http://127.1.49.2:46663/ using document root <none> and password file <none>
I20260926 13:43:27.301774 17337 fs_manager.cc:359] Metadata directory not provided
I20260926 13:43:27.301883 17337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:43:27.306269 17337 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:43:27.309046 17356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:27.310667 17337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 13:43:27.310845 17337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "2a0cbf7482a14f849b33b09526a24eb8"
format_stamp: "Formatted at 2026-09-26 13:43:06 on dist-test-slave-1wns"
server_key: "2ca6bc6f72630d057c5ea662d3dbfc05"
server_key_iv: "cc62d2f51aad59b32870982ea7ec184f"
server_key_version: "encryptionkey@0"
I20260926 13:43:27.311472 17337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:43:27.336825 17337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:43:27.340561 17337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:43:27.340806 17337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:43:27.341478 17337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:43:27.342712 17363 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 13:43:27.347172 17363 ts_tablet_manager.cc:1852] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 13:43:27.347654 17363 ts_tablet_manager.cc:1905] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:43:27.352371 17363 ts_tablet_manager.cc:1918] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:43:27.352499 17363 log.cc:1196] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/aa8b95e38fee43f7a037dd4f6e8ff55f
I20260926 13:43:27.352910 17363 ts_tablet_manager.cc:1939] T aa8b95e38fee43f7a037dd4f6e8ff55f P 2a0cbf7482a14f849b33b09526a24eb8: Deleting consensus metadata
I20260926 13:43:27.353310 17337 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 13:43:27.353401 17337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.001s sys 0.000s
I20260926 13:43:27.353497 17337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:43:27.353603 17337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:27.363896 17337 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:46021
I20260926 13:43:27.363925 17470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:46021 every 8 connection(s)
I20260926 13:43:27.364970 17337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:43:27.372164 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 17337
I20260926 13:43:27.372665 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:44803
--local_ip_for_outbound_sockets=127.1.49.3
--tserver_master_addrs=127.1.49.62:35215
--webserver_port=35079
--webserver_interface=127.1.49.3
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:43:27 dist-test-slave-1wns krb5kdc[16564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430207, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:43:27.381634 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.367210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:45947 (local address 127.1.49.62:35215)
0926 13:43:27.367353 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:27.367358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:27.368472 (+ 1114us) server_negotiation.cc:407] Connection header received
0926 13:43:27.369467 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:27.369470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:27.369544 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:27.369630 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:43:27.371019 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.371570 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.373426 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.373707 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.373810 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.376755 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:43:27.376763 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:43:27.376769 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:43:27.376812 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:43:27.378913 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:27.379596 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:27.379604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:27.379607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:27.379721 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:27.380192 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:27.380199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:27.380203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:27.380497 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:43:27.380619 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:27.381201 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 13:43:27.381378 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:43:27.382512 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.367698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:45947)
0926 13:43:27.368298 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:27.368350 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:27.369195 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:27.369870 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:27.369877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:27.370242 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:43:27.370845 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.370861 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.371726 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.371729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.373257 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.373275 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.373898 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.373905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.374172 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.374927 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:43:27.374944 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:43:27.376519 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:43:27.379092 (+ 2573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:27.379100 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:27.379119 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:27.379461 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:27.379898 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:27.379901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:27.379903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:27.380070 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:27.380620 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:43:27.380625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:43:27.380920 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 13:43:27.381093 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 13:43:27.381426 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":132,"threads_started":1}
I20260926 13:43:27.384034 17471 heartbeater.cc:347] Connected to a master server at 127.1.49.62:35215
I20260926 13:43:27.384316 17471 heartbeater.cc:464] Registering TS with master...
I20260926 13:43:27.384859 17471 heartbeater.cc:511] Master 127.1.49.62:35215 requested a full tablet report, sending...
I20260926 13:43:27.386011 17144 ts_manager.cc:194] Registered new tserver with Master: 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021)
I20260926 13:43:27.386689 17144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:45947
W20260926 13:43:27.497620 17474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:43:27.497977 17474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:43:27.498026 17474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:43:27.502353 17474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:43:27.502442 17474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:43:27.502508 17474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:43:27.508373 17474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:44803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=35079
--tserver_master_addrs=127.1.49.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:43:27.510968 17474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:43:27.512468 17474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:43:27.524170 17479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:27.524174 17482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:27.524212 17480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:27.525028 17474 server_base.cc:1034] running on GCE node
I20260926 13:43:27.525612 17474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:43:27.526496 17474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:43:27.527695 17474 hybrid_clock.cc:648] HybridClock initialized: now 1790430207527645 us; error 67 us; skew 500 ppm
Sep 26 13:43:27 dist-test-slave-1wns krb5kdc[16564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430207, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:43:27.531396 17474 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:43:27.532666 17474 webserver.cc:460] Webserver started at http://127.1.49.3:35079/ using document root <none> and password file <none>
I20260926 13:43:27.533309 17474 fs_manager.cc:359] Metadata directory not provided
I20260926 13:43:27.533433 17474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:43:27.537850 17474 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:43:27.540449 17489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:27.541884 17474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:43:27.542021 17474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "29ec3485bf1a4b61bc9bdb727aa67972"
format_stamp: "Formatted at 2026-09-26 13:43:06 on dist-test-slave-1wns"
server_key: "53a9da82a9942033ed3948db6aeae1a4"
server_key_iv: "74c4d3e149fbab79ae4996f3ad223de7"
server_key_version: "encryptionkey@0"
I20260926 13:43:27.542609 17474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:43:27.571236 17474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:43:27.575003 17474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:43:27.575242 17474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:43:27.575930 17474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:43:27.576822 17474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:43:27.576870 17474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:27.576937 17474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:43:27.576972 17474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:27.586557 17474 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:44803
I20260926 13:43:27.586576 17602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:44803 every 8 connection(s)
I20260926 13:43:27.587756 17474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:43:27.591355 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 17474
Sep 26 13:43:27 dist-test-slave-1wns krb5kdc[16564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430207, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:43:27.602592 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.592231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.1.49.62:35215)
0926 13:43:27.592498 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:27.592512 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:27.592680 (+ 168us) server_negotiation.cc:407] Connection header received
0926 13:43:27.592854 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:27.592858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:27.592907 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:27.593005 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:43:27.593998 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.594808 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.595745 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.596069 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.596192 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.598064 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:43:27.598067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:43:27.598069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:43:27.598097 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:43:27.600129 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:27.600711 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:27.600715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:27.600717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:27.600820 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:27.601478 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:27.601483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:27.601487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:27.601804 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:43:27.602146 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:27.602311 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 13:43:27.602428 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
I20260926 13:43:27.604856 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.589846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:59155 (local address 127.1.49.62:35215)
0926 13:43:27.590033 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:27.590037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:27.591548 (+ 1511us) server_negotiation.cc:407] Connection header received
0926 13:43:27.592788 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:27.592791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:27.592843 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:27.592922 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:43:27.594638 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.595139 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.596429 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.596790 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.596909 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.599919 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:43:27.599924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:43:27.599927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:43:27.599976 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:43:27.601973 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:27.602768 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:27.602772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:27.602775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:27.602889 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:27.603411 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:27.603415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:27.603416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:27.603765 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:43:27.603942 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:27.604527 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 13:43:27.604664 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 13:43:27.606103 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.590254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:59155)
0926 13:43:27.591345 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:27.591412 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:27.592545 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:27.593120 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:27.593127 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:27.593602 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:43:27.594418 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.594439 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.595284 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.595288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.596297 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.596308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.596979 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.596984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.597279 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.597956 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:43:27.597981 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:43:27.599659 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:43:27.602113 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:27.602122 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:27.602142 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:27.602612 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:27.603032 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:27.603036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:27.603038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:27.603245 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:27.603949 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:43:27.603956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:43:27.604263 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 13:43:27.604451 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 13:43:27.604828 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":595,"threads_started":1}
I20260926 13:43:27.607522 17603 heartbeater.cc:347] Connected to a master server at 127.1.49.62:35215
I20260926 13:43:27.607854 17603 heartbeater.cc:464] Registering TS with master...
I20260926 13:43:27.608445 17603 heartbeater.cc:511] Master 127.1.49.62:35215 requested a full tablet report, sending...
I20260926 13:43:27.609822 17144 ts_manager.cc:194] Registered new tserver with Master: 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803)
I20260926 13:43:27.610395 17144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:59155
I20260926 13:43:27.617653 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:43:27.622495 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.618409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37970 (local address 127.1.49.62:35215)
0926 13:43:27.618548 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:27.618551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:27.618853 (+ 302us) server_negotiation.cc:407] Connection header received
0926 13:43:27.618989 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:27.618991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:27.619112 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:27.619216 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:43:27.619475 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.620160 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.621152 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.621382 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.621470 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.621931 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:43:27.622113 (+ 182us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 13:43:27.622373 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37970: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 13:43:27.622602 17334 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37970: token expired
W20260926 13:43:27.623070 17074 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.49.62:35215)
I20260926 13:43:27.632620 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.623496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37980 (local address 127.1.49.62:35215)
0926 13:43:27.623685 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:27.623688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:27.623700 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:43:27.623733 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:27.623734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:27.623773 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:27.623918 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:43:27.624666 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.625176 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.626176 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.626592 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.626698 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.628357 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:43:27.628360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:43:27.628362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:43:27.628389 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:43:27.630201 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:27.630765 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:27.630768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:27.630769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:27.630854 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:27.631319 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:27.631322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:27.631323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:27.631625 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:43:27.631759 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:27.632355 (+ 596us) server_negotiation.cc:299] Negotiation successful
0926 13:43:27.632478 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260926 13:43:27.635937 17144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:43:27.639696 17144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:43:27.649299 17266 tablet_service.cc:1467] Processing CreateTablet for tablet 59da6d2d286b427c8145e919a6b803e7 (DEFAULT_TABLE table=test-table [id=87754fc3cde04c7a829ef67e6c85379f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:43:27.650029 17266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59da6d2d286b427c8145e919a6b803e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:43:27.653365 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.647866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:46021 (local address 127.0.0.1:58286)
0926 13:43:27.647976 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:27.647991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:27.648072 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:27.649078 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:27.649080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:27.649097 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:43:27.649209 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.649214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.650945 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.650948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.651971 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.652015 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.652892 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.652896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.653008 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.653069 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 13:43:27.653122 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 13:43:27.653213 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":94}
I20260926 13:43:27.653785 17615 tablet_bootstrap.cc:492] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192: Bootstrap starting.
I20260926 13:43:27.653947 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.648224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:44803 (local address 127.0.0.1:57648)
0926 13:43:27.648826 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:27.648838 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:27.648958 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:27.649535 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:27.649539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:27.649558 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:43:27.649659 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.649666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.650969 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.650972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.652195 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.652205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.653528 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.653531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.653638 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.653663 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 13:43:27.653735 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 13:43:27.653821 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260926 13:43:27.654186 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.648133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.1.49.2:46021)
0926 13:43:27.648521 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:27.648533 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:27.648560 (+ 27us) server_negotiation.cc:407] Connection header received
0926 13:43:27.648665 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:27.648669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:27.648913 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:27.649092 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:43:27.649339 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.650821 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.652131 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.652741 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.653653 (+ 912us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.653800 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:27.653888 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 13:43:27.654013 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":173,"threads_started":1}
I20260926 13:43:27.654361 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.648341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.1.49.3:44803)
0926 13:43:27.648676 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:27.648689 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:27.648931 (+ 242us) server_negotiation.cc:407] Connection header received
0926 13:43:27.649141 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:27.649146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:27.649384 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:27.649537 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:43:27.649803 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.650830 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.652338 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.653190 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.653297 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.653466 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:27.654122 (+ 656us) server_negotiation.cc:299] Negotiation successful
0926 13:43:27.654225 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":94,"threads_started":1}
I20260926 13:43:27.655655 17615 tablet_bootstrap.cc:654] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192: Neither blocks nor log segments found. Creating new log.
I20260926 13:43:27.655586 17405 tablet_service.cc:1467] Processing CreateTablet for tablet 59da6d2d286b427c8145e919a6b803e7 (DEFAULT_TABLE table=test-table [id=87754fc3cde04c7a829ef67e6c85379f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:43:27.656821 17405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59da6d2d286b427c8145e919a6b803e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:43:27.656970 17615 tablet_bootstrap.cc:492] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192: No bootstrap required, opened a new log
I20260926 13:43:27.657065 17615 ts_tablet_manager.cc:1397] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:43:27.657506 17615 raft_consensus.cc:348] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:27.657642 17615 raft_consensus.cc:374] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:43:27.657681 17615 raft_consensus.cc:729] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d243d45394354f9896d873e1e41f6192, State: Initialized, Role: FOLLOWER
I20260926 13:43:27.657332 17537 tablet_service.cc:1467] Processing CreateTablet for tablet 59da6d2d286b427c8145e919a6b803e7 (DEFAULT_TABLE table=test-table [id=87754fc3cde04c7a829ef67e6c85379f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:43:27.657820 17615 consensus_queue.cc:260] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:27.658162 17615 ts_tablet_manager.cc:1428] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 13:43:27.658404 17537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59da6d2d286b427c8145e919a6b803e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:43:27.661861 17363 tablet_bootstrap.cc:492] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8: Bootstrap starting.
I20260926 13:43:27.663992 17363 tablet_bootstrap.cc:654] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8: Neither blocks nor log segments found. Creating new log.
I20260926 13:43:27.664899 17363 log.cc:824] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:43:27.666808 17619 tablet_bootstrap.cc:492] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972: Bootstrap starting.
I20260926 13:43:27.667273 17363 tablet_bootstrap.cc:492] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8: No bootstrap required, opened a new log
I20260926 13:43:27.667491 17363 ts_tablet_manager.cc:1397] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 13:43:27.669057 17619 tablet_bootstrap.cc:654] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972: Neither blocks nor log segments found. Creating new log.
I20260926 13:43:27.670475 17619 log.cc:824] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972: Log is configured to *not* fsync() on all Append() calls
I20260926 13:43:27.671316 17363 raft_consensus.cc:348] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:27.671571 17363 raft_consensus.cc:374] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:43:27.671663 17363 raft_consensus.cc:729] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0cbf7482a14f849b33b09526a24eb8, State: Initialized, Role: FOLLOWER
I20260926 13:43:27.672228 17363 consensus_queue.cc:260] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:27.672794 17619 tablet_bootstrap.cc:492] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972: No bootstrap required, opened a new log
I20260926 13:43:27.673013 17619 ts_tablet_manager.cc:1397] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260926 13:43:27.673039 17471 heartbeater.cc:503] Master 127.1.49.62:35215 was elected leader, sending a full tablet report...
I20260926 13:43:27.673300 17363 ts_tablet_manager.cc:1428] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 13:43:27.676932 17619 raft_consensus.cc:348] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:27.677294 17619 raft_consensus.cc:374] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:43:27.677356 17619 raft_consensus.cc:729] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ec3485bf1a4b61bc9bdb727aa67972, State: Initialized, Role: FOLLOWER
I20260926 13:43:27.677780 17619 consensus_queue.cc:260] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:27.678457 17603 heartbeater.cc:503] Master 127.1.49.62:35215 was elected leader, sending a full tablet report...
I20260926 13:43:27.678821 17619 ts_tablet_manager.cc:1428] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260926 13:43:27.838869 17604 tablet.cc:2369] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:43:27.858337 17621 raft_consensus.cc:482] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:43:27.858546 17621 raft_consensus.cc:504] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:27.859762 17621 leader_election.cc:288] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803), d243d45394354f9896d873e1e41f6192 (127.1.49.1:44575)
I20260926 13:43:27.864383 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.860211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:44803 (local address 127.1.49.2:43697)
0926 13:43:27.860344 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:27.860362 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:27.860475 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:27.860823 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:27.860826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:27.860848 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:43:27.860975 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.860984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.862097 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.862100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.863102 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.863112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.863843 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.863847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.864007 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.864054 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 13:43:27.864140 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 13:43:27.864228 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":74}
I20260926 13:43:27.864408 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.860309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:43697 (local address 127.1.49.3:44803)
0926 13:43:27.860548 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:27.860553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:27.860579 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:43:27.860646 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:27.860649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:27.860699 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:27.860807 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:43:27.861138 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.861945 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.863245 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.863670 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.863799 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.863886 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:27.864175 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 13:43:27.864263 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260926 13:43:27.865314 17557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59da6d2d286b427c8145e919a6b803e7" candidate_uuid: "2a0cbf7482a14f849b33b09526a24eb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" is_pre_election: true
I20260926 13:43:27.865674 17557 raft_consensus.cc:2454] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a0cbf7482a14f849b33b09526a24eb8 in term 0.
I20260926 13:43:27.865793 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.860635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:44575 (local address 127.1.49.2:60925)
0926 13:43:27.861006 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:27.861018 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:27.861158 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:27.861459 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:27.861462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:27.861480 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:43:27.861613 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.861619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.862547 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.862550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.863945 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:27.863957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.865323 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.865327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:27.865438 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.865465 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 13:43:27.865556 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 13:43:27.865629 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":4,"thread_start_us":86,"threads_started":1}
I20260926 13:43:27.866118 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.860691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:60925 (local address 127.1.49.1:44575)
0926 13:43:27.861057 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:27.861064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:27.861111 (+ 47us) server_negotiation.cc:407] Connection header received
0926 13:43:27.861265 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:27.861267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:27.861322 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:27.861414 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:43:27.861748 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.862405 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.864084 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.865181 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.865293 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.865610 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:27.865652 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 13:43:27.865750 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":92,"threads_started":1}
I20260926 13:43:27.866302 17360 leader_election.cc:302] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29ec3485bf1a4b61bc9bdb727aa67972, 2a0cbf7482a14f849b33b09526a24eb8; no voters:
W20260926 13:43:27.866501 17472 tablet.cc:2369] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:43:27.866677 17621 raft_consensus.cc:2781] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:43:27.866842 17621 raft_consensus.cc:482] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:43:27.866823 17286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59da6d2d286b427c8145e919a6b803e7" candidate_uuid: "2a0cbf7482a14f849b33b09526a24eb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d243d45394354f9896d873e1e41f6192" is_pre_election: true
I20260926 13:43:27.866945 17621 raft_consensus.cc:3037] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:43:27.867120 17286 raft_consensus.cc:2454] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a0cbf7482a14f849b33b09526a24eb8 in term 0.
I20260926 13:43:27.868484 17621 raft_consensus.cc:504] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:27.868911 17621 leader_election.cc:288] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [CANDIDATE]: Term 1 election: Requested vote from peers 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803), d243d45394354f9896d873e1e41f6192 (127.1.49.1:44575)
I20260926 13:43:27.869314 17557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59da6d2d286b427c8145e919a6b803e7" candidate_uuid: "2a0cbf7482a14f849b33b09526a24eb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ec3485bf1a4b61bc9bdb727aa67972"
I20260926 13:43:27.869367 17286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59da6d2d286b427c8145e919a6b803e7" candidate_uuid: "2a0cbf7482a14f849b33b09526a24eb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d243d45394354f9896d873e1e41f6192"
I20260926 13:43:27.869508 17286 raft_consensus.cc:3037] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:43:27.869500 17557 raft_consensus.cc:3037] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:43:27.870666 17557 raft_consensus.cc:2454] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a0cbf7482a14f849b33b09526a24eb8 in term 1.
I20260926 13:43:27.870820 17286 raft_consensus.cc:2454] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a0cbf7482a14f849b33b09526a24eb8 in term 1.
I20260926 13:43:27.871068 17360 leader_election.cc:302] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29ec3485bf1a4b61bc9bdb727aa67972, 2a0cbf7482a14f849b33b09526a24eb8; no voters:
I20260926 13:43:27.871251 17621 raft_consensus.cc:2781] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:43:27.871573 17621 raft_consensus.cc:686] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [term 1 LEADER]: Becoming Leader. State: Replica: 2a0cbf7482a14f849b33b09526a24eb8, State: Running, Role: LEADER
I20260926 13:43:27.872087 17621 consensus_queue.cc:237] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:27.875327 17143 catalog_manager.cc:5495] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2). New cstate: current_term: 1 leader_uuid: "2a0cbf7482a14f849b33b09526a24eb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } health_report { overall_health: UNKNOWN } } }
W20260926 13:43:27.876205 17333 tablet.cc:2369] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:43:27.899369 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:27.895449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.1.49.2:46021)
0926 13:43:27.895602 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:27.895607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:27.895630 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:43:27.895851 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:27.895855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:27.895929 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:27.896111 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:43:27.896371 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.896883 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.897767 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:27.897999 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:27.898128 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:27.898455 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:43:27.898919 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:43:27.899080 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:27.899141 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 13:43:27.899207 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 13:43:27.906338 17286 raft_consensus.cc:1261] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Refusing update from remote peer 2a0cbf7482a14f849b33b09526a24eb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:43:27.906338 17557 raft_consensus.cc:1261] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972 [term 1 FOLLOWER]: Refusing update from remote peer 2a0cbf7482a14f849b33b09526a24eb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:43:27.907322 17625 consensus_queue.cc:1035] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:43:27.907572 17621 consensus_queue.cc:1035] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:43:27.916252 17630 mvcc.cc:205] Tried to move back new op lower bound from 7333602131574607872 to 7333602131446784000. Current Snapshot: MvccSnapshot[applied={T|T < 7333602131574607872}]
I20260926 13:43:27.925387 17144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:43:27.925630 17144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:43:27.928963 17144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Sending DeleteTablet for 3 replicas of tablet 59da6d2d286b427c8145e919a6b803e7
I20260926 13:43:27.929894 17405 tablet_service.cc:1514] Processing DeleteTablet for tablet 59da6d2d286b427c8145e919a6b803e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:43:27 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:58286
I20260926 13:43:27.930011 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 17182
I20260926 13:43:27.930200 17537 tablet_service.cc:1514] Processing DeleteTablet for tablet 59da6d2d286b427c8145e919a6b803e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:43:27 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:57648
I20260926 13:43:27.930841 17639 tablet_replica.cc:330] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972: stopping tablet replica
I20260926 13:43:27.931309 17639 raft_consensus.cc:2229] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:43:27.931747 17639 raft_consensus.cc:2258] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:43:27.932054 17638 tablet_replica.cc:330] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8: stopping tablet replica
I20260926 13:43:27.932406 17638 raft_consensus.cc:2229] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:43:27.932901 17638 raft_consensus.cc:2258] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:43:27.933892 17639 ts_tablet_manager.cc:1905] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:43:27.934450 17638 ts_tablet_manager.cc:1905] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:43:27.937350 17639 ts_tablet_manager.cc:1918] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:43:27.937393 17638 ts_tablet_manager.cc:1918] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:43:27.937474 17638 log.cc:1196] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/59da6d2d286b427c8145e919a6b803e7
I20260926 13:43:27.937651 17639 log.cc:1196] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/59da6d2d286b427c8145e919a6b803e7
I20260926 13:43:27.937855 17638 ts_tablet_manager.cc:1939] T 59da6d2d286b427c8145e919a6b803e7 P 2a0cbf7482a14f849b33b09526a24eb8: Deleting consensus metadata
I20260926 13:43:27.938208 17639 ts_tablet_manager.cc:1939] T 59da6d2d286b427c8145e919a6b803e7 P 29ec3485bf1a4b61bc9bdb727aa67972: Deleting consensus metadata
W20260926 13:43:27.938745 17131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:44575 (error 108)
I20260926 13:43:27.939560 17129 catalog_manager.cc:4841] TS 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021): tablet 59da6d2d286b427c8145e919a6b803e7 (table test-table [id=87754fc3cde04c7a829ef67e6c85379f]) successfully deleted
I20260926 13:43:27.939637 17131 catalog_manager.cc:4841] TS 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803): tablet 59da6d2d286b427c8145e919a6b803e7 (table test-table [id=87754fc3cde04c7a829ef67e6c85379f]) successfully deleted
W20260926 13:43:27.940326 17613 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:43:27.939971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:44575 (local address 127.0.0.1:43752)
0926 13:43:27.940179 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:27.940249 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:44575: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 13:43:27.940526 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 17337
W20260926 13:43:27.940552 17131 catalog_manager.cc:4584] TS d243d45394354f9896d873e1e41f6192 (127.1.49.1:44575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59da6d2d286b427c8145e919a6b803e7: Network error: Client connection negotiation failed: client connection to 127.1.49.1:44575: connect: Connection reset by peer (error 104)
I20260926 13:43:27.948872 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 17474
I20260926 13:43:27.954878 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 17111
I20260926 13:43:47.962667 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:35215
--webserver_interface=127.1.49.62
--webserver_port=35829
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:35215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:43:48.079298 17641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:43:48.079613 17641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:43:48.079703 17641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:43:48.083792 17641 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 13:43:48.083912 17641 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 13:43:48.083940 17641 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 13:43:48.083983 17641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:43:48.084045 17641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:43:48.084091 17641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:43:48.084127 17641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:43:48.084162 17641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:43:48.089946 17641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:35215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:35215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:43:48.091125 17641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:43:48.092211 17641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:43:48.102240 17641 server_base.cc:1034] running on GCE node
W20260926 13:43:48.102185 17649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:48.102185 17646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:48.102197 17647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:48.103204 17641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:43:48.104615 17641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:43:48.105813 17641 hybrid_clock.cc:648] HybridClock initialized: now 1790430228105736 us; error 89 us; skew 500 ppm
Sep 26 13:43:48 dist-test-slave-1wns krb5kdc[16564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430228, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:43:48.109464 17641 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:43:48.110528 17641 webserver.cc:460] Webserver started at http://127.1.49.62:35829/ using document root <none> and password file <none>
I20260926 13:43:48.111162 17641 fs_manager.cc:359] Metadata directory not provided
I20260926 13:43:48.111233 17641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:43:48.115711 17641 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:43:48.118251 17656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:48.119907 17641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:43:48.120067 17641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "1d9464aefe6143ba95a526c67b41fcbc"
format_stamp: "Formatted at 2026-09-26 13:43:05 on dist-test-slave-1wns"
server_key: "e1d1599f48ed981ed347132813c400b0"
server_key_iv: "5d9eeddf6d91cc3cc1201a95ae0b451a"
server_key_version: "encryptionkey@0"
I20260926 13:43:48.120694 17641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:43:48.137857 17641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:43:48.141439 17641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:43:48.141682 17641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:43:48.149690 17641 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:35215
I20260926 13:43:48.150153 17708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:35215 every 8 connection(s)
I20260926 13:43:48.150887 17641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:43:48.155552 17709 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:43:48.157630 17709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Bootstrap starting.
I20260926 13:43:48.160547 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 17641
I20260926 13:43:48.161139 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:44575
--local_ip_for_outbound_sockets=127.1.49.1
--tserver_master_addrs=127.1.49.62:35215
--webserver_port=44655
--webserver_interface=127.1.49.1
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:43:48.178407 17709 log.cc:824] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Log is configured to *not* fsync() on all Append() calls
I20260926 13:43:48.192024 17709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:43:48.192479 17709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: Bootstrap complete.
I20260926 13:43:48.197145 17709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } }
I20260926 13:43:48.197639 17709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9464aefe6143ba95a526c67b41fcbc, State: Initialized, Role: FOLLOWER
I20260926 13:43:48.198272 17709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } }
I20260926 13:43:48.198455 17709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:43:48.198547 17709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:43:48.198683 17709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 2 FOLLOWER]: Advancing to term 3
I20260926 13:43:48.201082 17709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } }
I20260926 13:43:48.201471 17709 leader_election.cc:302] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d9464aefe6143ba95a526c67b41fcbc; no voters:
I20260926 13:43:48.201854 17709 leader_election.cc:288] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 13:43:48.201967 17713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 3 FOLLOWER]: Leader election won for term 3
I20260926 13:43:48.202185 17713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [term 3 LEADER]: Becoming Leader. State: Replica: 1d9464aefe6143ba95a526c67b41fcbc, State: Running, Role: LEADER
I20260926 13:43:48.202586 17713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } }
I20260926 13:43:48.203269 17709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:43:48.204282 17715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d9464aefe6143ba95a526c67b41fcbc. Latest consensus state: current_term: 3 leader_uuid: "1d9464aefe6143ba95a526c67b41fcbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } } }
I20260926 13:43:48.204435 17715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: This master's current role is: LEADER
I20260926 13:43:48.204348 17714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1d9464aefe6143ba95a526c67b41fcbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d9464aefe6143ba95a526c67b41fcbc" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35215 } } }
I20260926 13:43:48.204491 17714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc [sys.catalog]: This master's current role is: LEADER
I20260926 13:43:48.209710 17729 catalog_manager.cc:1259] Loaded cluster ID: 4dc8bb7e2ae34d789a0dd74ba5aee739
I20260926 13:43:48.209802 17729 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: loading cluster ID for follower catalog manager: success
I20260926 13:43:48.212752 17729 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d9464aefe6143ba95a526c67b41fcbc: acquiring CA information for follower catalog manager: success
F20260926 13:43:48.214205 17729 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fba9d3f59ad google::LogMessage::Fail() at ??:0
@ 0x7fba9d3f888a google::LogMessage::SendToLog() at ??:0
@ 0x7fba9d3f54de google::LogMessage::Flush() at ??:0
@ 0x7fba9d3f6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbaa00b918d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbaa7d5e989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fbaa7d3b023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fbaa7d3b54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fbaa7d34652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fbaa7d339e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7fbaa7d7a541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fba9f638512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbaa0c7c6db start_thread at ??:0
@ 0x7fba9bef361f clone at ??:0
W20260926 13:43:48.304968 17711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:43:48.305261 17711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:43:48.305307 17711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:43:48.309589 17711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:43:48.309669 17711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:43:48.309736 17711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:43:48.315395 17711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:44575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=44655
--tserver_master_addrs=127.1.49.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:43:48.316648 17711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:43:48.317768 17711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:43:48.328414 17752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:48.328414 17749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:48.328954 17711 server_base.cc:1034] running on GCE node
W20260926 13:43:48.328454 17750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:48.329689 17711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:43:48.330470 17711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:43:48.331703 17711 hybrid_clock.cc:648] HybridClock initialized: now 1790430228331670 us; error 54 us; skew 500 ppm
Sep 26 13:43:48 dist-test-slave-1wns krb5kdc[16564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430228, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:43:48.335074 17711 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:43:48.336189 17711 webserver.cc:460] Webserver started at http://127.1.49.1:44655/ using document root <none> and password file <none>
I20260926 13:43:48.336867 17711 fs_manager.cc:359] Metadata directory not provided
I20260926 13:43:48.336975 17711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:43:48.341739 17711 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:43:48.344521 17759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:48.346032 17711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:43:48.346194 17711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "d243d45394354f9896d873e1e41f6192"
format_stamp: "Formatted at 2026-09-26 13:43:05 on dist-test-slave-1wns"
server_key: "70e96571770c9f8b9fe0a433815b8cfc"
server_key_iv: "775a960c23d6f0f4e1e4737887b2829a"
server_key_version: "encryptionkey@0"
I20260926 13:43:48.346820 17711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:43:48.374795 17711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:43:48.378871 17711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:43:48.379305 17711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:43:48.380097 17711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:43:48.381426 17766 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:43:48.387315 17711 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:43:48.387430 17711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260926 13:43:48.387521 17711 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:43:48.389958 17711 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:43:48.390071 17711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260926 13:43:48.390367 17766 tablet_bootstrap.cc:492] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192: Bootstrap starting.
I20260926 13:43:48.402390 17711 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:44575
I20260926 13:43:48.402694 17873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:44575 every 8 connection(s)
I20260926 13:43:48.403620 17711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:43:48.409394 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 17711
I20260926 13:43:48.409922 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:46021
--local_ip_for_outbound_sockets=127.1.49.2
--tserver_master_addrs=127.1.49.62:35215
--webserver_port=46663
--webserver_interface=127.1.49.2
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:43:48.413941 17766 log.cc:824] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192: Log is configured to *not* fsync() on all Append() calls
I20260926 13:43:48.424803 17766 tablet_bootstrap.cc:492] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:43:48.425351 17766 tablet_bootstrap.cc:492] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192: Bootstrap complete.
I20260926 13:43:48.426952 17766 ts_tablet_manager.cc:1397] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192: Time spent bootstrapping tablet: real 0.037s user 0.028s sys 0.004s
I20260926 13:43:48.430756 17766 raft_consensus.cc:348] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:48.431411 17766 raft_consensus.cc:729] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d243d45394354f9896d873e1e41f6192, State: Initialized, Role: FOLLOWER
I20260926 13:43:48.432075 17766 consensus_queue.cc:260] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:48.433413 17766 ts_tablet_manager.cc:1428] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
W20260926 13:43:48.444977 17876 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:43:48.405957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:41727)
0926 13:43:48.406550 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:48.406614 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:48.407615 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:48.444139 (+ 36524us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":173,"threads_started":1}
W20260926 13:43:48.445659 17874 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:35215 (0 consecutive failures): Network error: Failed to ping master at 127.1.49.62:35215: Client connection negotiation failed: client connection to 127.1.49.62:35215: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:43:48.446509 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:48.446195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:33897)
0926 13:43:48.446336 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:48.446438 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 13:43:48.447422 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:48.447164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:56271)
0926 13:43:48.447301 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:48.447363 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 13:43:48.447651 17874 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:43:48.536422 17877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:43:48.536762 17877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:43:48.536844 17877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:43:48.541235 17877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:43:48.541340 17877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:43:48.541441 17877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:43:48.547109 17877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:46021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=46663
--tserver_master_addrs=127.1.49.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:43:48.548348 17877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:43:48.549544 17877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:43:48.560294 17884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:48.560451 17877 server_base.cc:1034] running on GCE node
W20260926 13:43:48.560338 17885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:48.560345 17887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:48.561125 17877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:43:48.561841 17877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:43:48.563073 17877 hybrid_clock.cc:648] HybridClock initialized: now 1790430228563032 us; error 54 us; skew 500 ppm
Sep 26 13:43:48 dist-test-slave-1wns krb5kdc[16564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430228, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:43:48.566107 17877 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:43:48.567183 17877 webserver.cc:460] Webserver started at http://127.1.49.2:46663/ using document root <none> and password file <none>
I20260926 13:43:48.567925 17877 fs_manager.cc:359] Metadata directory not provided
I20260926 13:43:48.568020 17877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:43:48.572543 17877 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 13:43:48.575021 17894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:48.576565 17877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:43:48.576714 17877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "2a0cbf7482a14f849b33b09526a24eb8"
format_stamp: "Formatted at 2026-09-26 13:43:06 on dist-test-slave-1wns"
server_key: "2ca6bc6f72630d057c5ea662d3dbfc05"
server_key_iv: "cc62d2f51aad59b32870982ea7ec184f"
server_key_version: "encryptionkey@0"
I20260926 13:43:48.577325 17877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:43:48.592268 17877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:43:48.596407 17877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:43:48.596654 17877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:43:48.597429 17877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:43:48.598430 17877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:43:48.598481 17877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:48.598524 17877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:43:48.598564 17877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:48.609371 17877 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:46021
I20260926 13:43:48.609397 18007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:46021 every 8 connection(s)
I20260926 13:43:48.610468 17877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:43:48.614894 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:48.613123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:60511)
0926 13:43:48.613895 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:48.614100 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":123,"threads_started":1}
W20260926 13:43:48.615409 18008 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:35215 (0 consecutive failures): Network error: Failed to ping master at 127.1.49.62:35215: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
I20260926 13:43:48.616206 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:48.615959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:37479)
0926 13:43:48.616075 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:48.616136 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 13:43:48.616945 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:48.616709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:52479)
0926 13:43:48.616828 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:48.616878 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 13:43:48.617174 18008 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 13:43:48.620114 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 17877
I20260926 13:43:48.620500 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:44803
--local_ip_for_outbound_sockets=127.1.49.3
--tserver_master_addrs=127.1.49.62:35215
--webserver_port=35079
--webserver_interface=127.1.49.3
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:43:48.736943 18011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:43:48.737439 18011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:43:48.737535 18011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:43:48.743247 18011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:43:48.743336 18011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:43:48.743454 18011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:43:48.749497 18011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:44803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=35079
--tserver_master_addrs=127.1.49.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:43:48.750624 18011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:43:48.751758 18011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:43:48.761853 18017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:43:48.761874 18016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:48.762028 18011 server_base.cc:1034] running on GCE node
W20260926 13:43:48.761874 18019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:43:48.762565 18011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:43:48.763288 18011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:43:48.764501 18011 hybrid_clock.cc:648] HybridClock initialized: now 1790430228764484 us; error 34 us; skew 500 ppm
Sep 26 13:43:48 dist-test-slave-1wns krb5kdc[16564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430228, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:43:48.772444 18011 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:43:48.773545 18011 webserver.cc:460] Webserver started at http://127.1.49.3:35079/ using document root <none> and password file <none>
I20260926 13:43:48.774116 18011 fs_manager.cc:359] Metadata directory not provided
I20260926 13:43:48.774181 18011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:43:48.778354 18011 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:43:48.780726 18026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:48.782269 18011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:43:48.782382 18011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "29ec3485bf1a4b61bc9bdb727aa67972"
format_stamp: "Formatted at 2026-09-26 13:43:06 on dist-test-slave-1wns"
server_key: "53a9da82a9942033ed3948db6aeae1a4"
server_key_iv: "74c4d3e149fbab79ae4996f3ad223de7"
server_key_version: "encryptionkey@0"
I20260926 13:43:48.782990 18011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:43:48.795949 18011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:43:48.799207 18011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:43:48.799376 18011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:43:48.800024 18011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:43:48.800830 18011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:43:48.800879 18011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:48.800920 18011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:43:48.800936 18011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:43:48.811026 18011 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:44803
I20260926 13:43:48.811064 18139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:44803 every 8 connection(s)
I20260926 13:43:48.812228 18011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:43:48.816360 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:48.814564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:37887)
0926 13:43:48.815312 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:48.815459 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":122,"threads_started":1}
W20260926 13:43:48.816879 18140 heartbeater.cc:644] Failed to heartbeat to 127.1.49.62:35215 (0 consecutive failures): Network error: Failed to ping master at 127.1.49.62:35215: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
I20260926 13:43:48.817569 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:48.817317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:49533)
0926 13:43:48.817459 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:48.817511 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 13:43:48.817939 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 18011
I20260926 13:43:48.818578 18143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:48.818054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:43423)
0926 13:43:48.818430 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:48.818492 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
W20260926 13:43:48.818755 18140 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 13:43:49.448900 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:49.448417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:36861)
0926 13:43:49.448712 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:49.448782 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":80,"threads_started":1}
I20260926 13:43:49.618336 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:49.617847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:60883)
0926 13:43:49.618136 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:49.618219 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":75,"threads_started":1}
I20260926 13:43:49.670894 18147 raft_consensus.cc:482] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:43:49.671154 18147 raft_consensus.cc:504] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:49.672637 18147 leader_election.cc:288] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021), 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803)
Sep 26 13:43:49 dist-test-slave-1wns krb5kdc[16564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430228, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.2@KRBTEST.COM
Sep 26 13:43:49 dist-test-slave-1wns krb5kdc[16564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430228, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.3@KRBTEST.COM
I20260926 13:43:49.688689 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:49.672866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:46021 (local address 127.1.49.1:46099)
0926 13:43:49.673086 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:49.673104 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:49.673221 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:49.675166 (+ 1945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:49.675175 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:49.675783 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:43:49.676268 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:49.676284 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:49.677606 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:49.677613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:49.678587 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:49.678597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:49.679358 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:49.679362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:49.679595 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:49.680384 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:43:49.680396 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:43:49.682397 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:43:49.685391 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:49.685402 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:49.685420 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:49.686036 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:49.686517 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:49.686520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:49.686522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:49.686702 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:49.687547 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:43:49.687554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:43:49.687903 (+ 349us) client_negotiation.cc:769] Sending connection context
0926 13:43:49.688106 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 13:43:49.688474 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":100,"spinlock_wait_cycles":6016}
I20260926 13:43:49.688755 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:49.673092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:51759 (local address 127.1.49.3:44803)
0926 13:43:49.673525 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:49.673541 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:49.673586 (+ 45us) server_negotiation.cc:407] Connection header received
0926 13:43:49.674457 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:49.674465 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:49.674939 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:49.675266 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:43:49.676465 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:49.677543 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:49.678768 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:49.679236 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:49.679449 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:49.683050 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:43:49.683055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:43:49.683067 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:43:49.683098 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:43:49.685339 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:49.686361 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:49.686370 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:49.686379 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:49.686628 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:49.686957 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:49.686962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:49.686964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:49.687433 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:43:49.687585 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:49.688226 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 13:43:49.688575 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":91,"threads_started":1}
I20260926 13:43:49.688797 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:49.673069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:46099 (local address 127.1.49.2:46021)
0926 13:43:49.673393 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0926 13:43:49.673416 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 13:43:49.673456 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:43:49.674229 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:43:49.674240 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:43:49.674684 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:43:49.675033 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:43:49.676453 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:49.677452 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:49.678716 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:43:49.679220 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:43:49.679375 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:49.682681 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:43:49.682690 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:43:49.682710 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:43:49.682752 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:43:49.685226 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:49.686179 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:49.686201 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:49.686211 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:49.686383 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:43:49.686829 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:43:49.686832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:43:49.686834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:43:49.687414 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:43:49.687564 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 13:43:49.688226 (+ 662us) server_negotiation.cc:299] Negotiation successful
0926 13:43:49.688547 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260926 13:43:49.688736 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:49.672918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:44803 (local address 127.1.49.1:51759)
0926 13:43:49.673465 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:49.673479 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:43:49.673597 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:43:49.675224 (+ 1627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:43:49.675227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:43:49.675770 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:43:49.676268 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:49.676284 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:49.677698 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:49.677701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:49.678584 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:43:49.678594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:43:49.679397 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:43:49.679400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:43:49.679581 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:43:49.680351 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:43:49.680370 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:43:49.682854 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:43:49.685470 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:49.685475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:49.685477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:49.686208 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:49.686769 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:43:49.686775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:43:49.686780 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:43:49.686870 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:43:49.687570 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:43:49.687574 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:43:49.687905 (+ 331us) client_negotiation.cc:769] Sending connection context
0926 13:43:49.688106 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 13:43:49.688564 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":299,"spinlock_wait_cycles":43776,"thread_start_us":85,"threads_started":1}
W20260926 13:43:49.691438 17763 leader_election.cc:341] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803): Not found: Tablet not found: 59da6d2d286b427c8145e919a6b803e7
W20260926 13:43:49.691618 17761 leader_election.cc:341] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021): Not found: Tablet not found: 59da6d2d286b427c8145e919a6b803e7
I20260926 13:43:49.691716 17761 leader_election.cc:302] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d243d45394354f9896d873e1e41f6192; no voters: 29ec3485bf1a4b61bc9bdb727aa67972, 2a0cbf7482a14f849b33b09526a24eb8
I20260926 13:43:49.692025 18147 raft_consensus.cc:2726] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:43:49.819990 18151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:49.819431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:43617)
0926 13:43:49.819745 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:49.819875 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":97,"threads_started":1}
I20260926 13:43:50.450378 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:50.449869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:52053)
0926 13:43:50.450182 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:50.450250 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":82,"threads_started":1}
I20260926 13:43:50.619673 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:50.619226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:53033)
0926 13:43:50.619511 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:50.619574 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":100,"threads_started":1}
I20260926 13:43:50.821164 18154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:50.820735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:43869)
0926 13:43:50.820990 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:50.821051 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260926 13:43:51.451719 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:51.451302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:55497)
0926 13:43:51.451568 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:51.451623 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
I20260926 13:43:51.594154 18156 raft_consensus.cc:482] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:43:51.594312 18156 raft_consensus.cc:504] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:51.594807 18156 leader_election.cc:288] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021), 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803)
W20260926 13:43:51.595920 17763 leader_election.cc:341] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803): Not found: Tablet not found: 59da6d2d286b427c8145e919a6b803e7
W20260926 13:43:51.596055 17761 leader_election.cc:341] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021): Not found: Tablet not found: 59da6d2d286b427c8145e919a6b803e7
I20260926 13:43:51.596129 17761 leader_election.cc:302] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d243d45394354f9896d873e1e41f6192; no voters: 29ec3485bf1a4b61bc9bdb727aa67972, 2a0cbf7482a14f849b33b09526a24eb8
I20260926 13:43:51.596316 18156 raft_consensus.cc:2726] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:43:51.621268 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:51.620704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:37951)
0926 13:43:51.621086 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:51.621154 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":85,"threads_started":1}
I20260926 13:43:51.822739 18158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:51.822232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:47305)
0926 13:43:51.822563 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:51.822627 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":84,"threads_started":1}
I20260926 13:43:52.453171 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:52.452683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:55539)
0926 13:43:52.452980 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:52.453047 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":84,"threads_started":1}
I20260926 13:43:52.622572 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:52.622145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:59467)
0926 13:43:52.622419 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:52.622475 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":82,"threads_started":1}
I20260926 13:43:52.824329 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:52.823691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:58015)
0926 13:43:52.824129 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:52.824211 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":154,"threads_started":1}
I20260926 13:43:53.454833 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:53.454305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:36001)
0926 13:43:53.454634 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:53.454701 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":80,"threads_started":1}
I20260926 13:43:53.624310 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:53.623725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:38583)
0926 13:43:53.624073 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:53.624143 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":153,"threads_started":1}
I20260926 13:43:53.825862 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:53.825373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:48449)
0926 13:43:53.825686 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:53.825761 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
I20260926 13:43:54.096246 18165 raft_consensus.cc:482] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:43:54.096411 18165 raft_consensus.cc:504] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:54.096961 18165 leader_election.cc:288] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021), 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803)
W20260926 13:43:54.098011 17761 leader_election.cc:341] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021): Not found: Tablet not found: 59da6d2d286b427c8145e919a6b803e7
W20260926 13:43:54.098248 17763 leader_election.cc:341] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803): Not found: Tablet not found: 59da6d2d286b427c8145e919a6b803e7
I20260926 13:43:54.098345 17763 leader_election.cc:302] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d243d45394354f9896d873e1e41f6192; no voters: 29ec3485bf1a4b61bc9bdb727aa67972, 2a0cbf7482a14f849b33b09526a24eb8
I20260926 13:43:54.098577 18165 raft_consensus.cc:2726] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:43:54.456262 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:54.455786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:55995)
0926 13:43:54.456107 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:54.456164 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":86,"threads_started":1}
I20260926 13:43:54.625711 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:54.625167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:48005)
0926 13:43:54.625513 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:54.625578 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":142,"threads_started":1}
I20260926 13:43:54.827481 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:54.826947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:44597)
0926 13:43:54.827292 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:54.827370 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":83,"threads_started":1}
I20260926 13:43:55.457643 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:55.457113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:41491)
0926 13:43:55.457475 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:55.457536 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":142,"threads_started":1}
I20260926 13:43:55.627138 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:55.626625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:56927)
0926 13:43:55.626969 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:55.627034 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":85,"threads_started":1}
I20260926 13:43:55.829129 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:55.828618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:44537)
0926 13:43:55.828964 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:55.829031 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":81,"threads_started":1}
I20260926 13:43:56.459030 18172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:56.458565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:42481)
0926 13:43:56.458864 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:56.458929 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260926 13:43:56.628613 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:56.628128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:43161)
0926 13:43:56.628450 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:56.628512 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":110,"threads_started":1}
I20260926 13:43:56.830585 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:56.830045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:50699)
0926 13:43:56.830407 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:56.830472 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
I20260926 13:43:57.460850 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:57.460181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:39059)
0926 13:43:57.460550 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:57.460637 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":114,"threads_started":1}
I20260926 13:43:57.560223 18176 raft_consensus.cc:482] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:43:57.560367 18176 raft_consensus.cc:504] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:57.560834 18176 leader_election.cc:288] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021), 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803)
W20260926 13:43:57.561753 17763 leader_election.cc:341] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803): Not found: Tablet not found: 59da6d2d286b427c8145e919a6b803e7
W20260926 13:43:57.561901 17761 leader_election.cc:341] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021): Not found: Tablet not found: 59da6d2d286b427c8145e919a6b803e7
I20260926 13:43:57.562000 17761 leader_election.cc:302] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d243d45394354f9896d873e1e41f6192; no voters: 29ec3485bf1a4b61bc9bdb727aa67972, 2a0cbf7482a14f849b33b09526a24eb8
I20260926 13:43:57.562189 18176 raft_consensus.cc:2726] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:43:57.630074 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:57.629545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:36333)
0926 13:43:57.629898 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:57.629969 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":97,"threads_started":1}
I20260926 13:43:57.832110 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:57.831553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:58851)
0926 13:43:57.831937 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:57.832004 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":144,"threads_started":1}
I20260926 13:43:58.462473 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:58.461932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:53543)
0926 13:43:58.462241 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:58.462310 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":89,"threads_started":1}
I20260926 13:43:58.631656 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:58.631124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:39775)
0926 13:43:58.631461 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:58.631531 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":75,"threads_started":1}
I20260926 13:43:58.833696 18181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:58.833201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:37679)
0926 13:43:58.833519 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:58.833596 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":96,"threads_started":1}
I20260926 13:43:59.394542 18182 raft_consensus.cc:482] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:43:59.394723 18182 raft_consensus.cc:504] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a0cbf7482a14f849b33b09526a24eb8" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46021 } } peers { permanent_uuid: "29ec3485bf1a4b61bc9bdb727aa67972" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 44803 } } peers { permanent_uuid: "d243d45394354f9896d873e1e41f6192" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 44575 } }
I20260926 13:43:59.395249 18182 leader_election.cc:288] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021), 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803)
W20260926 13:43:59.396083 17761 leader_election.cc:341] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a0cbf7482a14f849b33b09526a24eb8 (127.1.49.2:46021): Not found: Tablet not found: 59da6d2d286b427c8145e919a6b803e7
W20260926 13:43:59.396209 17763 leader_election.cc:341] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 29ec3485bf1a4b61bc9bdb727aa67972 (127.1.49.3:44803): Not found: Tablet not found: 59da6d2d286b427c8145e919a6b803e7
I20260926 13:43:59.396347 17763 leader_election.cc:302] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d243d45394354f9896d873e1e41f6192; no voters: 29ec3485bf1a4b61bc9bdb727aa67972, 2a0cbf7482a14f849b33b09526a24eb8
I20260926 13:43:59.396508 18182 raft_consensus.cc:2726] T 59da6d2d286b427c8145e919a6b803e7 P d243d45394354f9896d873e1e41f6192 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:43:59.463969 18183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:59.463485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:56331)
0926 13:43:59.463764 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:59.463862 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260926 13:43:59.633065 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:59.632587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:48315)
0926 13:43:59.632896 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:59.632960 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":105,"threads_started":1}
I20260926 13:43:59.835091 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:43:59.834555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:59747)
0926 13:43:59.834910 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0926 13:43:59.834979 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":98,"threads_started":1}
I20260926 13:44:00.465621 18186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:00.465022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:45667)
0926 13:44:00.465378 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:00.465519 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
I20260926 13:44:00.634490 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:00.633951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:40865)
0926 13:44:00.634321 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:00.634388 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":82,"threads_started":1}
I20260926 13:44:00.837983 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:00.837244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:53789)
0926 13:44:00.837726 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:00.837804 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":84,"threads_started":1}
I20260926 13:44:01.467211 18189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:01.466612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:35899)
0926 13:44:01.466984 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:01.467067 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260926 13:44:01.636088 18190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:01.635441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:49739)
0926 13:44:01.635898 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:01.635966 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":92,"threads_started":1}
I20260926 13:44:01.839787 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:01.839089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:51089)
0926 13:44:01.839535 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:01.839630 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":134,"threads_started":1}
I20260926 13:44:02.468869 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:02.468385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:41107)
0926 13:44:02.468693 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:02.468765 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260926 13:44:02.637557 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:02.637076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:53919)
0926 13:44:02.637383 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:02.637454 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":94,"threads_started":1}
I20260926 13:44:02.841440 18194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:02.840923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.3:46927)
0926 13:44:02.841242 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:02.841325 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":142,"threads_started":1}
I20260926 13:44:03.470583 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:03.470101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.1:59501)
0926 13:44:03.470407 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:03.470475 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":97,"threads_started":1}
I20260926 13:44:03.639285 18196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:03.638796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:35215 (local address 127.1.49.2:45969)
0926 13:44:03.639116 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:03.639183 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.62:35215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":94,"threads_started":1}
I20260926 13:44:03.823379 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 17711
I20260926 13:44:03.830225 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 17877
I20260926 13:44:03.836181 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 18011
2026-09-26T13:44:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59687 ms)
[----------] 30 tests from SecurityITest (259989 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:03 dist-test-slave-1wns krb5kdc[18199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:03 dist-test-slave-1wns krb5kdc[18199](info): set up 2 sockets
Sep 26 13:44:03 dist-test-slave-1wns krb5kdc[18199](info): commencing operation
krb5kdc: starting...
W20260926 13:44:04.928351 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:05 dist-test-slave-1wns krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:44:05.068814 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:42829
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:42829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:05.189332 18212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:05.189706 18212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:05.189795 18212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:05.194069 18212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:05.194195 18212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:05.194231 18212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:05.194270 18212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:05.194305 18212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:05.200188 18212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:42829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:42829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:05.201442 18212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:05.202572 18212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:44:05.212931 18212 server_base.cc:1034] running on GCE node
W20260926 13:44:05.212888 18218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:05.212946 18220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:05.212880 18217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:05.213838 18212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:05.215140 18212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:05.216313 18212 hybrid_clock.cc:648] HybridClock initialized: now 1790430245216284 us; error 46 us; skew 500 ppm
Sep 26 13:44:05 dist-test-slave-1wns krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:05.219558 18212 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:44:05.220682 18212 webserver.cc:460] Webserver started at http://127.1.49.62:45129/ using document root <none> and password file <none>
I20260926 13:44:05.221305 18212 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:05.221370 18212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:05.221625 18212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:05.223948 18212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "a61a862a82284c14a48c104696223c76"
format_stamp: "Formatted at 2026-09-26 13:44:05 on dist-test-slave-1wns"
server_key: "bde61b5e01f65a50aa31cfd4bd3cd72e"
server_key_iv: "b2c10fa5e9cf89161f018b1f43324afa"
server_key_version: "encryptionkey@0"
I20260926 13:44:05.224592 18212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "a61a862a82284c14a48c104696223c76"
format_stamp: "Formatted at 2026-09-26 13:44:05 on dist-test-slave-1wns"
server_key: "bde61b5e01f65a50aa31cfd4bd3cd72e"
server_key_iv: "b2c10fa5e9cf89161f018b1f43324afa"
server_key_version: "encryptionkey@0"
I20260926 13:44:05.228986 18212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 13:44:05.232276 18227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:05.233959 18212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 13:44:05.234153 18212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "a61a862a82284c14a48c104696223c76"
format_stamp: "Formatted at 2026-09-26 13:44:05 on dist-test-slave-1wns"
server_key: "bde61b5e01f65a50aa31cfd4bd3cd72e"
server_key_iv: "b2c10fa5e9cf89161f018b1f43324afa"
server_key_version: "encryptionkey@0"
I20260926 13:44:05.234311 18212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:05.256618 18212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:05.260340 18212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:05.260578 18212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:05.268461 18212 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:42829
I20260926 13:44:05.268500 18279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:42829 every 8 connection(s)
I20260926 13:44:05.269594 18212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:05.273164 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:05.276963 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 18212
I20260926 13:44:05.277130 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:05.277493 1220 external_mini_cluster.cc:1428] Setting key 97cc31742bdc707a801be5fe9716fd04
I20260926 13:44:05.280520 18280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76: Bootstrap starting.
I20260926 13:44:05.283528 18280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:05.284474 18280 log.cc:824] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:05.286906 18280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76: No bootstrap required, opened a new log
Sep 26 13:44:05 dist-test-slave-1wns krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:05.290766 18280 raft_consensus.cc:348] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61a862a82284c14a48c104696223c76" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42829 } }
I20260926 13:44:05.291055 18280 raft_consensus.cc:374] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:05.291150 18280 raft_consensus.cc:729] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61a862a82284c14a48c104696223c76, State: Initialized, Role: FOLLOWER
I20260926 13:44:05.291680 18280 consensus_queue.cc:260] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61a862a82284c14a48c104696223c76" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42829 } }
I20260926 13:44:05.291863 18280 raft_consensus.cc:388] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:05.291956 18280 raft_consensus.cc:482] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:05.292078 18280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:05.293546 18280 raft_consensus.cc:504] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61a862a82284c14a48c104696223c76" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42829 } }
I20260926 13:44:05.294034 18280 leader_election.cc:302] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a61a862a82284c14a48c104696223c76; no voters:
I20260926 13:44:05.294497 18280 leader_election.cc:288] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:05.294682 18285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:05.294957 18285 raft_consensus.cc:686] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [term 1 LEADER]: Becoming Leader. State: Replica: a61a862a82284c14a48c104696223c76, State: Running, Role: LEADER
I20260926 13:44:05.295361 18285 consensus_queue.cc:237] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61a862a82284c14a48c104696223c76" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42829 } }
I20260926 13:44:05.296132 18280 sys_catalog.cc:564] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:05.296567 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:05.279139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.1.49.62:42829)
0926 13:44:05.279604 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:05.279632 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:05.279672 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:44:05.280479 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:05.280499 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:05.280961 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:05.281308 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:05.282430 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:05.283694 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:05.284813 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:05.285379 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:05.285596 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:05.288726 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:05.288732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:05.288749 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:05.288782 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:05.291569 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:05.292272 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:05.292278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:05.292284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:05.292414 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:05.292786 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:05.292792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:05.292794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:05.293577 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:05.293807 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:05.294271 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 13:44:05.294866 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":159,"threads_started":1}
I20260926 13:44:05.296808 18285 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a61a862a82284c14a48c104696223c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61a862a82284c14a48c104696223c76" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42829 } } }
I20260926 13:44:05.296929 18285 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:05.297226 18285 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a61a862a82284c14a48c104696223c76. Latest consensus state: current_term: 1 leader_uuid: "a61a862a82284c14a48c104696223c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61a862a82284c14a48c104696223c76" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42829 } } }
I20260926 13:44:05.297307 18285 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:05.297761 18291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:05.301354 18291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:05.308490 18291 catalog_manager.cc:1347] Generated new cluster ID: 4bc44f3e2ffc4cf59be52c289a92839c
I20260926 13:44:05.308583 18291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:05.321692 18291 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:05.322781 18291 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:05.330101 18291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76: Generated new TSK 0
I20260926 13:44:05.330967 18291 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:44:05.372128 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:42829
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:05.496781 18306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:05.497110 18306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:05.497159 18306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:05.501518 18306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:05.501592 18306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:05.501657 18306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:44:05.507190 18306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:42829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:05.508433 18306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:05.509637 18306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:05.520167 18311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:05.520215 18312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:05.520093 18314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:05.520452 18306 server_base.cc:1034] running on GCE node
I20260926 13:44:05.520975 18306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:05.521692 18306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:05.522917 18306 hybrid_clock.cc:648] HybridClock initialized: now 1790430245522872 us; error 54 us; skew 500 ppm
Sep 26 13:44:05 dist-test-slave-1wns krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:05.526189 18306 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:44:05.527148 18306 webserver.cc:460] Webserver started at http://127.1.49.1:32987/ using document root <none> and password file <none>
I20260926 13:44:05.527758 18306 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:05.527882 18306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:05.528127 18306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:05.530387 18306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "8b1a81248abd4376a3d2feec691294fb"
format_stamp: "Formatted at 2026-09-26 13:44:05 on dist-test-slave-1wns"
server_key: "0b958575e1218355ab1a0dfab974c4e3"
server_key_iv: "4746c37c284904564b5e29c510ccc20b"
server_key_version: "encryptionkey@0"
I20260926 13:44:05.531013 18306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "8b1a81248abd4376a3d2feec691294fb"
format_stamp: "Formatted at 2026-09-26 13:44:05 on dist-test-slave-1wns"
server_key: "0b958575e1218355ab1a0dfab974c4e3"
server_key_iv: "4746c37c284904564b5e29c510ccc20b"
server_key_version: "encryptionkey@0"
I20260926 13:44:05.535219 18306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 13:44:05.538163 18321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:05.539901 18306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:05.540048 18306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "8b1a81248abd4376a3d2feec691294fb"
format_stamp: "Formatted at 2026-09-26 13:44:05 on dist-test-slave-1wns"
server_key: "0b958575e1218355ab1a0dfab974c4e3"
server_key_iv: "4746c37c284904564b5e29c510ccc20b"
server_key_version: "encryptionkey@0"
I20260926 13:44:05.540179 18306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:05.556840 18306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:05.560703 18306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:05.560942 18306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:05.561705 18306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:05.562666 18306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:05.562713 18306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:05.562754 18306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:05.562770 18306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:05.572242 18306 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:39373
I20260926 13:44:05.572270 18434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:39373 every 8 connection(s)
I20260926 13:44:05.573199 18306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:44:05.580822 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 18306
I20260926 13:44:05.580973 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:44:05.581285 1220 external_mini_cluster.cc:1428] Setting key 21bfaf5fcb0ba97f813027d0935eeec9
Sep 26 13:44:05 dist-test-slave-1wns krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:05.589200 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:05.575406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:59623 (local address 127.1.49.62:42829)
0926 13:44:05.575605 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:05.575611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:05.576711 (+ 1100us) server_negotiation.cc:407] Connection header received
0926 13:44:05.578025 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:05.578028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:05.578088 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:05.578175 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:05.579515 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:05.580274 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:05.582096 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:05.582349 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:05.582476 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:05.585321 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:05.585324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:05.585326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:05.585353 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:05.586898 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:05.587485 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:05.587489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:05.587491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:05.587583 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:05.588011 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:05.588016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:05.588021 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:05.588214 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:05.588334 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:05.588883 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 13:44:05.589018 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 13:44:05.590096 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:05.575927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:42829 (local address 127.1.49.1:59623)
0926 13:44:05.576466 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:05.576576 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:05.577799 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:05.578349 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:05.578356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:05.578813 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:05.579330 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:05.579346 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:05.580439 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:05.580442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:05.581953 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:05.581964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:05.582602 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:05.582608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:05.582772 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:05.583499 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:05.583523 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:05.585122 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:05.587042 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:05.587049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:05.587068 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:05.587370 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:05.587680 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:05.587683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:05.587685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:05.587906 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:05.588344 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:05.588353 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:05.588633 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 13:44:05.588829 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 13:44:05.589217 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":143,"threads_started":1}
I20260926 13:44:05.591470 18435 heartbeater.cc:347] Connected to a master server at 127.1.49.62:42829
I20260926 13:44:05.591864 18435 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:05.592428 18435 heartbeater.cc:511] Master 127.1.49.62:42829 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
I20260926 13:44:05.594374 18244 ts_manager.cc:194] Registered new tserver with Master: 8b1a81248abd4376a3d2feec691294fb (127.1.49.1:39373)
I20260926 13:44:05.595741 18244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:59623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:44:05.616684 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:42829
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:05.745342 18440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:05.745651 18440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:05.745698 18440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:05.750269 18440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:05.750345 18440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:05.750407 18440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:44:05.756142 18440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:42829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:05.757337 18440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:05.758625 18440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:44:05.769855 18440 server_base.cc:1034] running on GCE node
W20260926 13:44:05.770087 18448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:05.770227 18446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:05.770238 18445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:05.770799 18440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:05.771490 18440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:05.772733 18440 hybrid_clock.cc:648] HybridClock initialized: now 1790430245772709 us; error 39 us; skew 500 ppm
Sep 26 13:44:05 dist-test-slave-1wns krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:05.776147 18440 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:44:05.777235 18440 webserver.cc:460] Webserver started at http://127.1.49.2:35045/ using document root <none> and password file <none>
I20260926 13:44:05.777879 18440 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:05.777951 18440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:05.778189 18440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:05.780640 18440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "034ff81af68d4d82a398288a900619a0"
format_stamp: "Formatted at 2026-09-26 13:44:05 on dist-test-slave-1wns"
server_key: "bc45f1c5e9269758fba031f9b2b7acba"
server_key_iv: "005a79ddde19858ad98963ef1acc312d"
server_key_version: "encryptionkey@0"
I20260926 13:44:05.781293 18440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "034ff81af68d4d82a398288a900619a0"
format_stamp: "Formatted at 2026-09-26 13:44:05 on dist-test-slave-1wns"
server_key: "bc45f1c5e9269758fba031f9b2b7acba"
server_key_iv: "005a79ddde19858ad98963ef1acc312d"
server_key_version: "encryptionkey@0"
I20260926 13:44:05.785727 18440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 13:44:05.788558 18455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:05.790004 18440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:05.790176 18440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "034ff81af68d4d82a398288a900619a0"
format_stamp: "Formatted at 2026-09-26 13:44:05 on dist-test-slave-1wns"
server_key: "bc45f1c5e9269758fba031f9b2b7acba"
server_key_iv: "005a79ddde19858ad98963ef1acc312d"
server_key_version: "encryptionkey@0"
I20260926 13:44:05.790318 18440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:05.806066 18440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:05.809958 18440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:05.810196 18440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:05.810904 18440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:05.811937 18440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:05.812011 18440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:05.812093 18440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:05.812140 18440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:05.822831 18440 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:37205
I20260926 13:44:05.822917 18568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:37205 every 8 connection(s)
I20260926 13:44:05.823949 18440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:44:05.824932 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 18440
I20260926 13:44:05.825081 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:44:05.825457 1220 external_mini_cluster.cc:1428] Setting key 966fdbefc30cbd72d18a1bd3989d8690
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
Sep 26 13:44:05 dist-test-slave-1wns krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:05.846776 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:05.826418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:58697 (local address 127.1.49.62:42829)
0926 13:44:05.826579 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:05.826584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:05.827684 (+ 1100us) server_negotiation.cc:407] Connection header received
0926 13:44:05.829450 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:05.829454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:05.829504 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:05.829661 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:05.831931 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:05.832601 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:05.834886 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:05.835156 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:05.835261 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:05.841777 (+ 6516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:05.841781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:05.841784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:05.841820 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:05.844070 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:05.844810 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:05.844815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:05.844817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:05.844939 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:05.845397 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:05.845403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:05.845405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:05.845648 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:05.845775 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:05.846403 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 13:44:05.846567 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 13:44:05.847750 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:05.826811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:42829 (local address 127.1.49.2:58697)
0926 13:44:05.827512 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:05.827563 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:05.829164 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:05.829934 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:05.829946 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:05.830699 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:05.831544 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:05.831573 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:05.832781 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:05.832787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:05.834679 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:05.834696 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:05.835366 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:05.835371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:05.835670 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:05.836434 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:05.836465 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:05.841419 (+ 4954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:05.844233 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:05.844242 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:05.844263 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:05.844643 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:05.845083 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:05.845087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:05.845090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:05.845276 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:05.845791 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:05.845802 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:05.846113 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 13:44:05.846353 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 13:44:05.846770 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":208,"threads_started":1}
I20260926 13:44:05.849395 18569 heartbeater.cc:347] Connected to a master server at 127.1.49.62:42829
I20260926 13:44:05.849720 18569 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:05.850380 18569 heartbeater.cc:511] Master 127.1.49.62:42829 requested a full tablet report, sending...
I20260926 13:44:05.851884 18244 ts_manager.cc:194] Registered new tserver with Master: 034ff81af68d4d82a398288a900619a0 (127.1.49.2:37205)
I20260926 13:44:05.852576 18244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:58697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:44:05.860095 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:42829
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:05.980818 18574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:05.981118 18574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:05.981165 18574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:05.985368 18574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:05.985446 18574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:05.985510 18574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:44:05.991166 18574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:42829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:05.992458 18574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:05.993549 18574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:06.004908 18582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:06.005002 18574 server_base.cc:1034] running on GCE node
W20260926 13:44:06.004897 18579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:06.004868 18580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:06.005744 18574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:06.006740 18574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:06.008013 18574 hybrid_clock.cc:648] HybridClock initialized: now 1790430246007971 us; error 55 us; skew 500 ppm
Sep 26 13:44:06 dist-test-slave-1wns krb5kdc[18199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430246, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:06.011652 18574 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:44:06.012866 18574 webserver.cc:460] Webserver started at http://127.1.49.3:44769/ using document root <none> and password file <none>
I20260926 13:44:06.013664 18574 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:06.013778 18574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:06.014061 18574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:06.016619 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c"
format_stamp: "Formatted at 2026-09-26 13:44:06 on dist-test-slave-1wns"
server_key: "e3b384f640bcbe996d3dc739735b8599"
server_key_iv: "a1e63f8a4bc5656fff36b4cad9c08f6f"
server_key_version: "encryptionkey@0"
I20260926 13:44:06.017326 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c"
format_stamp: "Formatted at 2026-09-26 13:44:06 on dist-test-slave-1wns"
server_key: "e3b384f640bcbe996d3dc739735b8599"
server_key_iv: "a1e63f8a4bc5656fff36b4cad9c08f6f"
server_key_version: "encryptionkey@0"
I20260926 13:44:06.022116 18574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:44:06.025032 18589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:06.026628 18574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:06.026794 18574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c"
format_stamp: "Formatted at 2026-09-26 13:44:06 on dist-test-slave-1wns"
server_key: "e3b384f640bcbe996d3dc739735b8599"
server_key_iv: "a1e63f8a4bc5656fff36b4cad9c08f6f"
server_key_version: "encryptionkey@0"
I20260926 13:44:06.026930 18574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:06.055302 18574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:06.059165 18574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:06.059402 18574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:06.060117 18574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:06.061009 18574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:06.061084 18574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:06.061154 18574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:06.061206 18574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:06.070719 18574 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:42809
I20260926 13:44:06.070809 18702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:42809 every 8 connection(s)
I20260926 13:44:06.071744 18574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:44:06.078410 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 18574
I20260926 13:44:06.078579 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:44:06.078884 1220 external_mini_cluster.cc:1428] Setting key c999aedc6a9694b34717ed135971afb3
Sep 26 13:44:06 dist-test-slave-1wns krb5kdc[18199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430246, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:06.087720 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.073980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:51157 (local address 127.1.49.62:42829)
0926 13:44:06.074211 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:06.074217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:06.075284 (+ 1067us) server_negotiation.cc:407] Connection header received
0926 13:44:06.076669 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:06.076672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:06.076725 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:06.076814 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:06.078236 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.078893 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.080407 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.080630 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.080715 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.083499 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:06.083502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:06.083504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:06.083531 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:06.085362 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:06.085948 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:06.085952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:06.085953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:06.086045 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:06.086481 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:06.086483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:06.086485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:06.086762 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:06.086848 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:06.087451 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 13:44:06.087591 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 13:44:06.089156 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.074461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:42829 (local address 127.1.49.3:51157)
0926 13:44:06.075073 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:06.075123 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:06.076424 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:06.077050 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:06.077058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:06.077456 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:06.077982 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.077996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.079054 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.079063 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.080261 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.080272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.080787 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.080791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.081008 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.081671 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:06.081688 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:06.083332 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:06.085497 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:06.085503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:06.085518 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:06.085809 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:06.086172 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:06.086176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:06.086177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:06.086359 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:06.086877 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:06.086882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:06.087175 (+ 293us) client_negotiation.cc:769] Sending connection context
0926 13:44:06.087379 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 13:44:06.087773 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":135,"threads_started":1}
I20260926 13:44:06.090517 18703 heartbeater.cc:347] Connected to a master server at 127.1.49.62:42829
I20260926 13:44:06.090838 18703 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:06.091449 18703 heartbeater.cc:511] Master 127.1.49.62:42829 requested a full tablet report, sending...
I20260926 13:44:06.092774 18244 ts_manager.cc:194] Registered new tserver with Master: ffd3a1b6d0914c57a04fd5f865b7e19c (127.1.49.3:42809)
I20260926 13:44:06.093470 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:44:06.093595 18244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:51157
I20260926 13:44:06.105034 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.096100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.1.49.62:42829)
0926 13:44:06.096245 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:06.096249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:06.096522 (+ 273us) server_negotiation.cc:407] Connection header received
0926 13:44:06.096629 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:06.096631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:06.096672 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:06.096742 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:06.097561 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.098150 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.099299 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.099532 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.099711 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.101262 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:06.101265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:06.101267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:06.101293 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:06.102874 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:06.103459 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:06.103462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:06.103463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:06.103547 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:06.103984 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:06.103988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:06.103989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:06.104245 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:06.104373 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:06.104598 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 13:44:06.104830 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:44:06.108786 18244 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:44:06.112324 18244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:44:06.129170 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.122480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:42809 (local address 127.0.0.1:58516)
0926 13:44:06.123538 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:06.123562 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:06.123705 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:06.124435 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:06.124441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:06.124466 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:06.124628 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.124635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.126205 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.126208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.127507 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.127514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.128496 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.128499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.128622 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.128668 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 13:44:06.128812 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 13:44:06.128973 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":160,"spinlock_wait_cycles":770432,"thread_start_us":111,"threads_started":1}
I20260926 13:44:06.129272 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.122359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.1.49.2:37205)
0926 13:44:06.123445 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:06.123457 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:06.123874 (+ 417us) server_negotiation.cc:407] Connection header received
0926 13:44:06.124035 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:06.124040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:06.124242 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:06.124422 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:06.124929 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.126052 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.127367 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.128340 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.128462 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.128623 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:06.128988 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 13:44:06.129121 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":120,"threads_started":1}
I20260926 13:44:06.129662 18717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.122250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:39373 (local address 127.0.0.1:59802)
0926 13:44:06.123928 (+ 1678us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:06.123943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:06.124024 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:06.124604 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:06.124607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:06.124621 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:06.124756 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.124768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.125805 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.125808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.127323 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.127335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.129235 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.129241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.129355 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.129389 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 13:44:06.129458 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:44:06.129529 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":8,"spinlock_wait_cycles":912512,"thread_start_us":102,"threads_started":1}
I20260926 13:44:06.129170 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.122197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:37205 (local address 127.0.0.1:60456)
0926 13:44:06.123504 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:06.123562 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:06.123759 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:06.124626 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:06.124629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:06.124662 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:06.124788 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.124796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.126204 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.126208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.127224 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.127235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.128721 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.128724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.128834 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.128854 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 13:44:06.128922 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 13:44:06.128972 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":34,"spinlock_wait_cycles":9472,"thread_start_us":105,"threads_started":1}
I20260926 13:44:06.129932 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.123044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.1.49.3:42809)
0926 13:44:06.123657 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:06.123669 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:06.123696 (+ 27us) server_negotiation.cc:407] Connection header received
0926 13:44:06.124001 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:06.124007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:06.124242 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:06.124422 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:06.124758 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.126042 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.127633 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.128350 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.128490 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.128643 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:06.128875 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 13:44:06.129020 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":149,"threads_started":1}
I20260926 13:44:06.130103 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.123056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.1.49.1:39373)
0926 13:44:06.123467 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:06.123474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:06.124021 (+ 547us) server_negotiation.cc:407] Connection header received
0926 13:44:06.124145 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:06.124150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:06.124299 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:06.124456 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:06.124880 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.125633 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.127442 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.128340 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.128462 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.128622 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:06.129584 (+ 962us) server_negotiation.cc:299] Negotiation successful
0926 13:44:06.129700 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":120,"threads_started":1}
I20260926 13:44:06.132892 18637 tablet_service.cc:1467] Processing CreateTablet for tablet cb1419de32d94cefa313db2f48d4098d (DEFAULT_TABLE table=test-table [id=e670a68ab2c94441970a3d56b787b4cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:06.132886 18369 tablet_service.cc:1467] Processing CreateTablet for tablet cb1419de32d94cefa313db2f48d4098d (DEFAULT_TABLE table=test-table [id=e670a68ab2c94441970a3d56b787b4cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:06.132880 18503 tablet_service.cc:1467] Processing CreateTablet for tablet cb1419de32d94cefa313db2f48d4098d (DEFAULT_TABLE table=test-table [id=e670a68ab2c94441970a3d56b787b4cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:06.134145 18503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb1419de32d94cefa313db2f48d4098d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:06.134428 18637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb1419de32d94cefa313db2f48d4098d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:06.134428 18369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb1419de32d94cefa313db2f48d4098d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:06.141757 18720 tablet_bootstrap.cc:492] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0: Bootstrap starting.
I20260926 13:44:06.143265 18722 tablet_bootstrap.cc:492] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c: Bootstrap starting.
I20260926 13:44:06.143503 18721 tablet_bootstrap.cc:492] T cb1419de32d94cefa313db2f48d4098d P 8b1a81248abd4376a3d2feec691294fb: Bootstrap starting.
I20260926 13:44:06.144366 18720 tablet_bootstrap.cc:654] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:06.145714 18722 tablet_bootstrap.cc:654] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:06.145714 18721 tablet_bootstrap.cc:654] T cb1419de32d94cefa313db2f48d4098d P 8b1a81248abd4376a3d2feec691294fb: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:06.146018 18720 log.cc:824] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:06.146606 18721 log.cc:824] T cb1419de32d94cefa313db2f48d4098d P 8b1a81248abd4376a3d2feec691294fb: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:06.146893 18722 log.cc:824] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:06.148345 18720 tablet_bootstrap.cc:492] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0: No bootstrap required, opened a new log
I20260926 13:44:06.148568 18720 ts_tablet_manager.cc:1397] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 13:44:06.148622 18721 tablet_bootstrap.cc:492] T cb1419de32d94cefa313db2f48d4098d P 8b1a81248abd4376a3d2feec691294fb: No bootstrap required, opened a new log
I20260926 13:44:06.148770 18721 ts_tablet_manager.cc:1397] T cb1419de32d94cefa313db2f48d4098d P 8b1a81248abd4376a3d2feec691294fb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:44:06.149070 18722 tablet_bootstrap.cc:492] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c: No bootstrap required, opened a new log
I20260926 13:44:06.149253 18722 ts_tablet_manager.cc:1397] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:44:06.152259 18721 raft_consensus.cc:348] T cb1419de32d94cefa313db2f48d4098d P 8b1a81248abd4376a3d2feec691294fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034ff81af68d4d82a398288a900619a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37205 } } peers { permanent_uuid: "8b1a81248abd4376a3d2feec691294fb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 39373 } } peers { permanent_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42809 } }
I20260926 13:44:06.152529 18721 raft_consensus.cc:374] T cb1419de32d94cefa313db2f48d4098d P 8b1a81248abd4376a3d2feec691294fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:06.152608 18721 raft_consensus.cc:729] T cb1419de32d94cefa313db2f48d4098d P 8b1a81248abd4376a3d2feec691294fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1a81248abd4376a3d2feec691294fb, State: Initialized, Role: FOLLOWER
I20260926 13:44:06.153177 18721 consensus_queue.cc:260] T cb1419de32d94cefa313db2f48d4098d P 8b1a81248abd4376a3d2feec691294fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034ff81af68d4d82a398288a900619a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37205 } } peers { permanent_uuid: "8b1a81248abd4376a3d2feec691294fb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 39373 } } peers { permanent_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42809 } }
I20260926 13:44:06.153169 18722 raft_consensus.cc:348] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034ff81af68d4d82a398288a900619a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37205 } } peers { permanent_uuid: "8b1a81248abd4376a3d2feec691294fb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 39373 } } peers { permanent_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42809 } }
I20260926 13:44:06.153456 18722 raft_consensus.cc:374] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:06.153519 18722 raft_consensus.cc:729] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd3a1b6d0914c57a04fd5f865b7e19c, State: Initialized, Role: FOLLOWER
I20260926 13:44:06.153945 18722 consensus_queue.cc:260] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034ff81af68d4d82a398288a900619a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37205 } } peers { permanent_uuid: "8b1a81248abd4376a3d2feec691294fb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 39373 } } peers { permanent_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42809 } }
I20260926 13:44:06.153960 18720 raft_consensus.cc:348] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034ff81af68d4d82a398288a900619a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37205 } } peers { permanent_uuid: "8b1a81248abd4376a3d2feec691294fb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 39373 } } peers { permanent_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42809 } }
I20260926 13:44:06.154238 18720 raft_consensus.cc:374] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:06.154299 18720 raft_consensus.cc:729] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 034ff81af68d4d82a398288a900619a0, State: Initialized, Role: FOLLOWER
I20260926 13:44:06.154321 18721 ts_tablet_manager.cc:1428] T cb1419de32d94cefa313db2f48d4098d P 8b1a81248abd4376a3d2feec691294fb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:44:06.154740 18435 heartbeater.cc:503] Master 127.1.49.62:42829 was elected leader, sending a full tablet report...
I20260926 13:44:06.154707 18720 consensus_queue.cc:260] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034ff81af68d4d82a398288a900619a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37205 } } peers { permanent_uuid: "8b1a81248abd4376a3d2feec691294fb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 39373 } } peers { permanent_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42809 } }
I20260926 13:44:06.154882 18722 ts_tablet_manager.cc:1428] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:44:06.155224 18703 heartbeater.cc:503] Master 127.1.49.62:42829 was elected leader, sending a full tablet report...
I20260926 13:44:06.155423 18569 heartbeater.cc:503] Master 127.1.49.62:42829 was elected leader, sending a full tablet report...
I20260926 13:44:06.156661 18720 ts_tablet_manager.cc:1428] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260926 13:44:06.166231 18727 raft_consensus.cc:482] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:44:06.166414 18727 raft_consensus.cc:504] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034ff81af68d4d82a398288a900619a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37205 } } peers { permanent_uuid: "8b1a81248abd4376a3d2feec691294fb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 39373 } } peers { permanent_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42809 } }
I20260926 13:44:06.167562 18727 leader_election.cc:288] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 034ff81af68d4d82a398288a900619a0 (127.1.49.2:37205), 8b1a81248abd4376a3d2feec691294fb (127.1.49.1:39373)
I20260926 13:44:06.172569 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.167886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:37205 (local address 127.1.49.3:32803)
0926 13:44:06.168011 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:06.168028 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:06.168127 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:06.168412 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:06.168415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:06.168439 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:06.168580 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.168589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.169682 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.169684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.171166 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.171177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.172082 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.172085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.172204 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.172234 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 13:44:06.172324 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 13:44:06.172403 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":70}
I20260926 13:44:06.172560 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.168007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:32803 (local address 127.1.49.2:37205)
0926 13:44:06.168136 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:06.168176 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:06.168197 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:44:06.168266 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:06.168268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:06.168308 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:06.168376 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:06.168691 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.169574 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.171298 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.171951 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.172065 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.172130 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:06.172345 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 13:44:06.172419 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:44:06.172610 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.168044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:36521 (local address 127.1.49.1:39373)
0926 13:44:06.168160 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:06.168164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:06.168518 (+ 354us) server_negotiation.cc:407] Connection header received
0926 13:44:06.168595 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:06.168597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:06.168675 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:06.168754 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:06.168973 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.169697 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.171382 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.172027 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.172149 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.172213 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:06.172413 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 13:44:06.172493 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 13:44:06.172631 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.167894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:39373 (local address 127.1.49.3:36521)
0926 13:44:06.168413 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:06.168425 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:06.168515 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:06.168771 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:06.168773 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:06.168788 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:06.168870 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.168873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.169795 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.169798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.171167 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.171177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.172153 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.172157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.172284 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.172302 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 13:44:06.172395 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 13:44:06.172468 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":219,"thread_start_us":132,"threads_started":1}
I20260926 13:44:06.173326 18523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1419de32d94cefa313db2f48d4098d" candidate_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "034ff81af68d4d82a398288a900619a0" is_pre_election: true
I20260926 13:44:06.173574 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1419de32d94cefa313db2f48d4098d" candidate_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b1a81248abd4376a3d2feec691294fb" is_pre_election: true
I20260926 13:44:06.173663 18523 raft_consensus.cc:2454] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd3a1b6d0914c57a04fd5f865b7e19c in term 0.
I20260926 13:44:06.173867 18389 raft_consensus.cc:2454] T cb1419de32d94cefa313db2f48d4098d P 8b1a81248abd4376a3d2feec691294fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd3a1b6d0914c57a04fd5f865b7e19c in term 0.
I20260926 13:44:06.174212 18592 leader_election.cc:302] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 034ff81af68d4d82a398288a900619a0, ffd3a1b6d0914c57a04fd5f865b7e19c; no voters:
I20260926 13:44:06.174513 18727 raft_consensus.cc:2781] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:44:06.174610 18727 raft_consensus.cc:482] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:44:06.174670 18727 raft_consensus.cc:3037] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:06.175724 18727 raft_consensus.cc:504] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034ff81af68d4d82a398288a900619a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37205 } } peers { permanent_uuid: "8b1a81248abd4376a3d2feec691294fb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 39373 } } peers { permanent_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42809 } }
I20260926 13:44:06.176133 18727 leader_election.cc:288] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [CANDIDATE]: Term 1 election: Requested vote from peers 034ff81af68d4d82a398288a900619a0 (127.1.49.2:37205), 8b1a81248abd4376a3d2feec691294fb (127.1.49.1:39373)
I20260926 13:44:06.176540 18523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1419de32d94cefa313db2f48d4098d" candidate_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "034ff81af68d4d82a398288a900619a0"
I20260926 13:44:06.176549 18389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1419de32d94cefa313db2f48d4098d" candidate_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b1a81248abd4376a3d2feec691294fb"
I20260926 13:44:06.176653 18523 raft_consensus.cc:3037] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:06.176687 18389 raft_consensus.cc:3037] T cb1419de32d94cefa313db2f48d4098d P 8b1a81248abd4376a3d2feec691294fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:06.177742 18523 raft_consensus.cc:2454] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd3a1b6d0914c57a04fd5f865b7e19c in term 1.
I20260926 13:44:06.177802 18389 raft_consensus.cc:2454] T cb1419de32d94cefa313db2f48d4098d P 8b1a81248abd4376a3d2feec691294fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd3a1b6d0914c57a04fd5f865b7e19c in term 1.
I20260926 13:44:06.178107 18592 leader_election.cc:302] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 034ff81af68d4d82a398288a900619a0, ffd3a1b6d0914c57a04fd5f865b7e19c; no voters:
I20260926 13:44:06.178309 18727 raft_consensus.cc:2781] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:06.178546 18727 raft_consensus.cc:686] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [term 1 LEADER]: Becoming Leader. State: Replica: ffd3a1b6d0914c57a04fd5f865b7e19c, State: Running, Role: LEADER
I20260926 13:44:06.178911 18727 consensus_queue.cc:237] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034ff81af68d4d82a398288a900619a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37205 } } peers { permanent_uuid: "8b1a81248abd4376a3d2feec691294fb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 39373 } } peers { permanent_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42809 } }
I20260926 13:44:06.183159 18244 catalog_manager.cc:5495] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c reported cstate change: term changed from 0 to 1, leader changed from <none> to ffd3a1b6d0914c57a04fd5f865b7e19c (127.1.49.3). New cstate: current_term: 1 leader_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034ff81af68d4d82a398288a900619a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b1a81248abd4376a3d2feec691294fb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 39373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffd3a1b6d0914c57a04fd5f865b7e19c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 42809 } health_report { overall_health: HEALTHY } } }
I20260926 13:44:06.209568 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.205729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.1.49.3:42809)
0926 13:44:06.205900 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:06.205905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:06.205921 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:44:06.206008 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:06.206011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:06.206065 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:06.206182 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:06.206439 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.207017 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.207978 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.208204 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.208343 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.208665 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:06.209095 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:06.209229 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:06.209290 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 13:44:06.209350 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 13:44:06.215977 18523 raft_consensus.cc:1261] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0 [term 1 FOLLOWER]: Refusing update from remote peer ffd3a1b6d0914c57a04fd5f865b7e19c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:06.215977 18389 raft_consensus.cc:1261] T cb1419de32d94cefa313db2f48d4098d P 8b1a81248abd4376a3d2feec691294fb [term 1 FOLLOWER]: Refusing update from remote peer ffd3a1b6d0914c57a04fd5f865b7e19c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:06.216909 18727 consensus_queue.cc:1035] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [LEADER]: Connected to new peer: Peer: permanent_uuid: "034ff81af68d4d82a398288a900619a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:06.217144 18730 consensus_queue.cc:1035] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b1a81248abd4376a3d2feec691294fb" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 39373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:06.228606 18735 mvcc.cc:205] Tried to move back new op lower bound from 7333602288491917312 to 7333602288351604736. Current Snapshot: MvccSnapshot[applied={T|T < 7333602288491917312}]
I20260926 13:44:06.234191 18241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:06.234392 18241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:06.237319 18241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a61a862a82284c14a48c104696223c76: Sending DeleteTablet for 3 replicas of tablet cb1419de32d94cefa313db2f48d4098d
I20260926 13:44:06.238296 18369 tablet_service.cc:1514] Processing DeleteTablet for tablet cb1419de32d94cefa313db2f48d4098d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:06 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:59802
I20260926 13:44:06.238343 18503 tablet_service.cc:1514] Processing DeleteTablet for tablet cb1419de32d94cefa313db2f48d4098d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:06 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:60456
I20260926 13:44:06.238479 18637 tablet_service.cc:1514] Processing DeleteTablet for tablet cb1419de32d94cefa313db2f48d4098d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:06 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:58516
I20260926 13:44:06.238902 18742 tablet_replica.cc:330] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0: stopping tablet replica
I20260926 13:44:06.239252 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 18306
I20260926 13:44:06.240530 18742 raft_consensus.cc:2229] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:44:06.241025 18742 raft_consensus.cc:2258] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:06.242846 18742 ts_tablet_manager.cc:1905] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:06.245894 18742 ts_tablet_manager.cc:1918] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:44:06.246042 18742 log.cc:1196] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/cb1419de32d94cefa313db2f48d4098d
I20260926 13:44:06.246356 18744 tablet_replica.cc:330] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c: stopping tablet replica
I20260926 13:44:06.246411 18742 ts_tablet_manager.cc:1939] T cb1419de32d94cefa313db2f48d4098d P 034ff81af68d4d82a398288a900619a0: Deleting consensus metadata
I20260926 13:44:06.246840 18744 raft_consensus.cc:2229] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:44:06.247273 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 18440
W20260926 13:44:06.247273 18231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:39373 (error 108)
I20260926 13:44:06.247416 18744 raft_consensus.cc:2258] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:06.250057 18744 ts_tablet_manager.cc:1905] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:06.252467 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.248548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:39373 (local address 127.0.0.1:59804)
0926 13:44:06.252256 (+ 3708us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:06.252355 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:39373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3597}
W20260926 13:44:06.252676 18231 catalog_manager.cc:4584] TS 8b1a81248abd4376a3d2feec691294fb (127.1.49.1:39373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb1419de32d94cefa313db2f48d4098d: Network error: Client connection negotiation failed: client connection to 127.1.49.1:39373: connect: Connection refused (error 111)
I20260926 13:44:06.253434 18744 ts_tablet_manager.cc:1918] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:44:06.253540 18744 log.cc:1196] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/cb1419de32d94cefa313db2f48d4098d
I20260926 13:44:06.253888 18744 ts_tablet_manager.cc:1939] T cb1419de32d94cefa313db2f48d4098d P ffd3a1b6d0914c57a04fd5f865b7e19c: Deleting consensus metadata
I20260926 13:44:06.255400 18228 catalog_manager.cc:4841] TS ffd3a1b6d0914c57a04fd5f865b7e19c (127.1.49.3:42809): tablet cb1419de32d94cefa313db2f48d4098d (table test-table [id=e670a68ab2c94441970a3d56b787b4cd]) successfully deleted
I20260926 13:44:06.256202 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 18574
I20260926 13:44:06.256392 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.256102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:37205 (local address 127.0.0.1:60462)
0926 13:44:06.256242 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:06.256302 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.2:37205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 13:44:06.261720 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 18212
2026-09-26T13:44:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2424 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T13:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:06Z Disabled control of system clock
I20260926 13:44:06.292865 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:36381
--webserver_interface=127.1.49.62
--webserver_port=0
--builtin_ntp_servers=127.1.49.20:33861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:36381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:44:06.411253 18749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:06.411742 18749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:06.411845 18749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:06.416291 18749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:06.416355 18749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:06.416370 18749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:06.416388 18749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:06.416404 18749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:06.421494 18749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:33861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:36381
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:36381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:06.422561 18749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:06.424012 18749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:06.432915 18755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:06.433040 18749 server_base.cc:1034] running on GCE node
W20260926 13:44:06.432935 18757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:06.432958 18754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:06.433847 18749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:06.435576 18749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:06.436828 18749 hybrid_clock.cc:648] HybridClock initialized: now 1790430246436794 us; error 56 us; skew 500 ppm
I20260926 13:44:06.439105 18749 webserver.cc:460] Webserver started at http://127.1.49.62:41933/ using document root <none> and password file <none>
I20260926 13:44:06.439761 18749 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:06.439888 18749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:06.440146 18749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:06.442363 18749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "0d60fe5e0bb04d35a97c6bc18f7a3341"
format_stamp: "Formatted at 2026-09-26 13:44:06 on dist-test-slave-1wns"
server_key: "3fc258a7069849cba55bddd47c1d11f4"
server_key_iv: "6af0bde2eeb670d3ee36d5fa2c83e0ee"
server_key_version: "encryptionkey@0"
I20260926 13:44:06.443051 18749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "0d60fe5e0bb04d35a97c6bc18f7a3341"
format_stamp: "Formatted at 2026-09-26 13:44:06 on dist-test-slave-1wns"
server_key: "3fc258a7069849cba55bddd47c1d11f4"
server_key_iv: "6af0bde2eeb670d3ee36d5fa2c83e0ee"
server_key_version: "encryptionkey@0"
I20260926 13:44:06.447883 18749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 13:44:06.450668 18763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:06.452163 18749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:06.452313 18749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "0d60fe5e0bb04d35a97c6bc18f7a3341"
format_stamp: "Formatted at 2026-09-26 13:44:06 on dist-test-slave-1wns"
server_key: "3fc258a7069849cba55bddd47c1d11f4"
server_key_iv: "6af0bde2eeb670d3ee36d5fa2c83e0ee"
server_key_version: "encryptionkey@0"
I20260926 13:44:06.452441 18749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:06.480959 18749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:06.481761 18749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:06.481983 18749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:06.489621 18749 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:36381
I20260926 13:44:06.489681 18815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:36381 every 8 connection(s)
I20260926 13:44:06.490753 18749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:06.494323 18816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:06.500072 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 18749
I20260926 13:44:06.500231 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:06.500519 1220 external_mini_cluster.cc:1428] Setting key 15e8728d2cb263e18f71f7fe56373bde
I20260926 13:44:06.500541 18816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341: Bootstrap starting.
I20260926 13:44:06.503358 18816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:06.504272 18816 log.cc:824] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:06.506556 18816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341: No bootstrap required, opened a new log
I20260926 13:44:06.510638 18816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d60fe5e0bb04d35a97c6bc18f7a3341" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36381 } }
I20260926 13:44:06.510921 18816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:06.511010 18816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d60fe5e0bb04d35a97c6bc18f7a3341, State: Initialized, Role: FOLLOWER
I20260926 13:44:06.511427 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.502253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59544 (local address 127.1.49.62:36381)
0926 13:44:06.503093 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:06.503110 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:06.503159 (+ 49us) server_negotiation.cc:407] Connection header received
0926 13:44:06.504010 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:06.504032 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:06.504530 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:06.504900 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:06.505368 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.506361 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.507342 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.507920 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.508144 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.509139 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:06.509146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:06.509161 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:06.509706 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:06.509770 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:06.509800 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:06.509932 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:06.510100 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 13:44:06.510404 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":142,"threads_started":1}
I20260926 13:44:06.511463 18816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d60fe5e0bb04d35a97c6bc18f7a3341" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36381 } }
I20260926 13:44:06.511657 18816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:06.511723 18816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:06.511843 18816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:06.512982 18816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d60fe5e0bb04d35a97c6bc18f7a3341" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36381 } }
I20260926 13:44:06.513334 18816 leader_election.cc:302] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d60fe5e0bb04d35a97c6bc18f7a3341; no voters:
I20260926 13:44:06.513674 18816 leader_election.cc:288] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:06.513749 18821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:06.513952 18821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [term 1 LEADER]: Becoming Leader. State: Replica: 0d60fe5e0bb04d35a97c6bc18f7a3341, State: Running, Role: LEADER
I20260926 13:44:06.514256 18821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d60fe5e0bb04d35a97c6bc18f7a3341" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36381 } }
I20260926 13:44:06.515040 18816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:06.515512 18821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d60fe5e0bb04d35a97c6bc18f7a3341" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d60fe5e0bb04d35a97c6bc18f7a3341" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36381 } } }
I20260926 13:44:06.515641 18821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:06.515872 18821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d60fe5e0bb04d35a97c6bc18f7a3341. Latest consensus state: current_term: 1 leader_uuid: "0d60fe5e0bb04d35a97c6bc18f7a3341" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d60fe5e0bb04d35a97c6bc18f7a3341" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 36381 } } }
I20260926 13:44:06.515978 18821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:06.516309 18825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:06.519600 18825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:06.525192 18825 catalog_manager.cc:1347] Generated new cluster ID: 23a0f979bbbe40edbecc153c64615459
I20260926 13:44:06.525275 18825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:06.544282 18825 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:06.545421 18825 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:06.553632 18825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341: Generated new TSK 0
I20260926 13:44:06.554471 18825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:06.560662 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36381
--builtin_ntp_servers=127.1.49.20:33861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:44:06.679608 18840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:06.679971 18840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:06.680018 18840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:06.684283 18840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:06.684355 18840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:06.684413 18840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:44:06.689556 18840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:33861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:06.690768 18840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:06.692304 18840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:06.702745 18846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:06.702793 18848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:06.702787 18845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:06.703249 18840 server_base.cc:1034] running on GCE node
I20260926 13:44:06.703696 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:06.704425 18840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:06.705592 18840 hybrid_clock.cc:648] HybridClock initialized: now 1790430246705577 us; error 29 us; skew 500 ppm
I20260926 13:44:06.707778 18840 webserver.cc:460] Webserver started at http://127.1.49.1:46645/ using document root <none> and password file <none>
I20260926 13:44:06.708452 18840 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:06.708515 18840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:06.708691 18840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:06.710757 18840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "6fd6a798769841cca8b46ac22dfd75f0"
format_stamp: "Formatted at 2026-09-26 13:44:06 on dist-test-slave-1wns"
server_key: "d8f3181ec8548857b62c6f0ed5cdbc88"
server_key_iv: "4bc8169883f0e030b7208520ee242993"
server_key_version: "encryptionkey@0"
I20260926 13:44:06.711267 18840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "6fd6a798769841cca8b46ac22dfd75f0"
format_stamp: "Formatted at 2026-09-26 13:44:06 on dist-test-slave-1wns"
server_key: "d8f3181ec8548857b62c6f0ed5cdbc88"
server_key_iv: "4bc8169883f0e030b7208520ee242993"
server_key_version: "encryptionkey@0"
I20260926 13:44:06.715317 18840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 13:44:06.718230 18854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:06.719722 18840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:44:06.719904 18840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "6fd6a798769841cca8b46ac22dfd75f0"
format_stamp: "Formatted at 2026-09-26 13:44:06 on dist-test-slave-1wns"
server_key: "d8f3181ec8548857b62c6f0ed5cdbc88"
server_key_iv: "4bc8169883f0e030b7208520ee242993"
server_key_version: "encryptionkey@0"
I20260926 13:44:06.720017 18840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:06.736610 18840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:06.737385 18840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:06.737584 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:06.738286 18840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:06.739369 18840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:06.739434 18840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:06.739522 18840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:06.739559 18840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:06.750027 18840 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:40019
I20260926 13:44:06.750049 18967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:40019 every 8 connection(s)
I20260926 13:44:06.751223 18840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:44:06.758406 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 18840
I20260926 13:44:06.758538 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:44:06.758832 1220 external_mini_cluster.cc:1428] Setting key f2d93234e27ea27d9c064524ffe796a2
I20260926 13:44:06.761212 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36381
--builtin_ntp_servers=127.1.49.20:33861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:44:06.763010 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.753417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:51155 (local address 127.1.49.62:36381)
0926 13:44:06.753605 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:06.753609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:06.754815 (+ 1206us) server_negotiation.cc:407] Connection header received
0926 13:44:06.756696 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:06.756703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:06.756778 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:06.756867 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:06.757976 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.758939 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.760483 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.760861 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.761002 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.762403 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:06.762410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:06.762416 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:06.762435 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:06.762464 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:06.762478 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:06.762596 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:06.762833 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 13:44:06.762893 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 13:44:06.764662 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.753857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36381 (local address 127.1.49.1:51155)
0926 13:44:06.754599 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:06.754657 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:06.756270 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:06.757139 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:06.757148 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:06.757251 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:06.757811 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.757827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.759112 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.759116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.760331 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.760346 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.761045 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.761049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.761219 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.762066 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:44:06.762068 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:06.762129 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:44:06.762216 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:06.762628 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:06.762640 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:06.762650 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 13:44:06.762828 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 13:44:06.763337 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":120,"threads_started":1}
I20260926 13:44:06.766253 18968 heartbeater.cc:347] Connected to a master server at 127.1.49.62:36381
I20260926 13:44:06.766537 18968 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:06.767220 18968 heartbeater.cc:511] Master 127.1.49.62:36381 requested a full tablet report, sending...
I20260926 13:44:06.768961 18780 ts_manager.cc:194] Registered new tserver with Master: 6fd6a798769841cca8b46ac22dfd75f0 (127.1.49.1:40019)
I20260926 13:44:06.770202 18780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.49.1:51155
W20260926 13:44:06.885936 18971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:06.886257 18971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:06.886332 18971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:06.890575 18971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:06.890686 18971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:06.890782 18971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:44:06.895690 18971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:33861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:06.896869 18971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:06.898312 18971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:06.908448 18979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:06.908458 18976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:06.908469 18977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:06.908825 18971 server_base.cc:1034] running on GCE node
I20260926 13:44:06.909356 18971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:06.910048 18971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:06.911263 18971 hybrid_clock.cc:648] HybridClock initialized: now 1790430246911242 us; error 37 us; skew 500 ppm
I20260926 13:44:06.913486 18971 webserver.cc:460] Webserver started at http://127.1.49.2:43933/ using document root <none> and password file <none>
I20260926 13:44:06.914137 18971 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:06.914237 18971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:06.914484 18971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:06.917065 18971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403"
format_stamp: "Formatted at 2026-09-26 13:44:06 on dist-test-slave-1wns"
server_key: "c48188ca75dcf07c59a894b90de107c3"
server_key_iv: "9fb527d1fb96b3ab60ade552b67ffcc1"
server_key_version: "encryptionkey@0"
I20260926 13:44:06.917732 18971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403"
format_stamp: "Formatted at 2026-09-26 13:44:06 on dist-test-slave-1wns"
server_key: "c48188ca75dcf07c59a894b90de107c3"
server_key_iv: "9fb527d1fb96b3ab60ade552b67ffcc1"
server_key_version: "encryptionkey@0"
I20260926 13:44:06.922129 18971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:44:06.925097 18985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:06.926563 18971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:06.926785 18971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403"
format_stamp: "Formatted at 2026-09-26 13:44:06 on dist-test-slave-1wns"
server_key: "c48188ca75dcf07c59a894b90de107c3"
server_key_iv: "9fb527d1fb96b3ab60ade552b67ffcc1"
server_key_version: "encryptionkey@0"
I20260926 13:44:06.926921 18971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:06.952456 18971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:06.953289 18971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:06.953521 18971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:06.954209 18971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:06.955653 18971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:06.955720 18971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:06.955789 18971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:06.955850 18971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 13:44:06.966146 18971 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:37445
I20260926 13:44:06.966192 19098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:37445 every 8 connection(s)
I20260926 13:44:06.967209 18971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:44:06.969916 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 18971
I20260926 13:44:06.970055 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:44:06.970381 1220 external_mini_cluster.cc:1428] Setting key eeaba2e05ff6da567382be9327cb2de9
I20260926 13:44:06.972878 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36381
--builtin_ntp_servers=127.1.49.20:33861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:44:06.979926 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.969638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:53085 (local address 127.1.49.62:36381)
0926 13:44:06.969794 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:06.969798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:06.971888 (+ 2090us) server_negotiation.cc:407] Connection header received
0926 13:44:06.973323 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:06.973327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:06.973386 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:06.973477 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:06.974569 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.975210 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.977270 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.977545 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.977731 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.979327 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:06.979330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:06.979333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:06.979348 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:06.979369 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:06.979377 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:06.979442 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:06.979668 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 13:44:06.979727 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 13:44:06.981411 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:06.970233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36381 (local address 127.1.49.2:53085)
0926 13:44:06.971164 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:06.971212 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:06.973063 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:06.973745 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:06.973754 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:06.973856 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:06.974409 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.974423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.975353 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.975360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.977125 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:06.977141 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:06.977823 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:06.977830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:06.978112 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:06.978925 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:44:06.978928 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:06.979003 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:44:06.979119 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:06.979477 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:06.979490 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:06.979499 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 13:44:06.979677 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 13:44:06.980115 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":157,"threads_started":1}
I20260926 13:44:06.982748 19099 heartbeater.cc:347] Connected to a master server at 127.1.49.62:36381
I20260926 13:44:06.983053 19099 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:06.983909 19099 heartbeater.cc:511] Master 127.1.49.62:36381 requested a full tablet report, sending...
I20260926 13:44:06.985409 18780 ts_manager.cc:194] Registered new tserver with Master: ccb43a5ac7824ebc8b62a07d4d9a7403 (127.1.49.2:37445)
I20260926 13:44:06.986361 18780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.49.2:53085
W20260926 13:44:07.098150 19102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:07.098513 19102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:07.098575 19102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:07.103055 19102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:07.103147 19102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:07.103210 19102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:44:07.108783 19102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:33861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:36381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:07.109942 19102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:07.111344 19102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:07.122478 19110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:07.122476 19108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:07.122483 19107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:07.122843 19102 server_base.cc:1034] running on GCE node
I20260926 13:44:07.123438 19102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:07.124279 19102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:07.125537 19102 hybrid_clock.cc:648] HybridClock initialized: now 1790430247125498 us; error 59 us; skew 500 ppm
I20260926 13:44:07.127720 19102 webserver.cc:460] Webserver started at http://127.1.49.3:45043/ using document root <none> and password file <none>
I20260926 13:44:07.128412 19102 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:07.128513 19102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:07.128767 19102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:07.131122 19102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "3958833f832544eab2dccb2a488e1098"
format_stamp: "Formatted at 2026-09-26 13:44:07 on dist-test-slave-1wns"
server_key: "5cddcb602bde7cd9ed0074d32c3be93b"
server_key_iv: "6f54345e54a838659718d1a84360a5a9"
server_key_version: "encryptionkey@0"
I20260926 13:44:07.131748 19102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "3958833f832544eab2dccb2a488e1098"
format_stamp: "Formatted at 2026-09-26 13:44:07 on dist-test-slave-1wns"
server_key: "5cddcb602bde7cd9ed0074d32c3be93b"
server_key_iv: "6f54345e54a838659718d1a84360a5a9"
server_key_version: "encryptionkey@0"
I20260926 13:44:07.136010 19102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:44:07.138847 19116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:07.140625 19102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:07.140873 19102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "3958833f832544eab2dccb2a488e1098"
format_stamp: "Formatted at 2026-09-26 13:44:07 on dist-test-slave-1wns"
server_key: "5cddcb602bde7cd9ed0074d32c3be93b"
server_key_iv: "6f54345e54a838659718d1a84360a5a9"
server_key_version: "encryptionkey@0"
I20260926 13:44:07.141021 19102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:07.157130 19102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:07.157958 19102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:07.158197 19102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:07.158900 19102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:07.160071 19102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:07.160147 19102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:07.160229 19102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:07.160262 19102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:07.170817 19102 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:39321
I20260926 13:44:07.170895 19229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:39321 every 8 connection(s)
I20260926 13:44:07.171977 19102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:44:07.181354 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 19102
I20260926 13:44:07.181474 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:44:07.181772 1220 external_mini_cluster.cc:1428] Setting key 76f7e14a01f456f3c72a5ef90611c311
I20260926 13:44:07.183190 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.174199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:52859 (local address 127.1.49.62:36381)
0926 13:44:07.174374 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:07.174380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:07.175606 (+ 1226us) server_negotiation.cc:407] Connection header received
0926 13:44:07.177228 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:07.177231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:07.177286 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:07.177376 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:07.178492 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.179101 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.180651 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.180912 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.181054 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:07.182381 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:07.182385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:07.182387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:07.182401 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:07.182423 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:07.182432 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:07.182522 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:07.182942 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 13:44:07.183024 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 13:44:07.184216 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.174668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:36381 (local address 127.1.49.3:52859)
0926 13:44:07.175424 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:07.175474 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:07.176967 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:07.177638 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:07.177646 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:07.177747 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:07.178316 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:07.178332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.179251 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.179254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:07.180372 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:07.180384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.181091 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.181094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:07.181266 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:07.182074 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:44:07.182076 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:07.182122 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:44:07.182195 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:07.182525 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:07.182538 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:07.182545 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:44:07.182785 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 13:44:07.183072 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":130,"threads_started":1}
I20260926 13:44:07.185456 19230 heartbeater.cc:347] Connected to a master server at 127.1.49.62:36381
I20260926 13:44:07.185792 19230 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:07.186363 19230 heartbeater.cc:511] Master 127.1.49.62:36381 requested a full tablet report, sending...
I20260926 13:44:07.187440 18780 ts_manager.cc:194] Registered new tserver with Master: 3958833f832544eab2dccb2a488e1098 (127.1.49.3:39321)
I20260926 13:44:07.188133 18780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.49.3:52859
I20260926 13:44:07.196830 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:44:07.203488 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.199166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.1.49.62:36381)
0926 13:44:07.199340 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:07.199345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:07.199486 (+ 141us) server_negotiation.cc:407] Connection header received
0926 13:44:07.199682 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:07.199684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:07.199738 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:07.199872 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:07.200157 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.200672 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.201587 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.201887 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.201987 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:07.202899 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:07.202902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:07.202905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:07.202917 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:07.202938 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:07.202947 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:07.203023 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:07.203196 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 13:44:07.203257 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:44:07.207015 18780 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:44:07.210291 18780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:44:07.225929 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.219985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53032 (local address 127.1.49.2:37445)
0926 13:44:07.220486 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:07.220500 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:07.221042 (+ 542us) server_negotiation.cc:407] Connection header received
0926 13:44:07.221138 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:07.221147 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:07.221389 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:07.221570 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:07.221908 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.222787 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.224482 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.225197 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.225333 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:07.225573 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:07.225644 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 13:44:07.225744 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":135,"threads_started":1}
I20260926 13:44:07.225922 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.219735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:37445 (local address 127.0.0.1:53032)
0926 13:44:07.220657 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:07.220714 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:07.220871 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:07.221568 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:07.221578 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:07.221601 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:07.221782 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:07.221789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.222915 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.222921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:07.224355 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:07.224365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.225330 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.225335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:07.225455 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:07.225498 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 13:44:07.225612 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 13:44:07.225738 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":97,"threads_started":1}
I20260926 13:44:07.226907 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.220060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:40019 (local address 127.0.0.1:58072)
0926 13:44:07.220904 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:07.220919 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:07.221010 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:07.221723 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:07.221726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:07.221743 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:07.221821 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:07.221831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.223678 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.223681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:07.225286 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:07.225295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.226459 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.226464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:07.226556 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:07.226592 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 13:44:07.226670 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 13:44:07.226755 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":29,"thread_start_us":139,"threads_started":1}
I20260926 13:44:07.226979 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.220550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.1.49.1:40019)
0926 13:44:07.221205 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:07.221213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:07.221240 (+ 27us) server_negotiation.cc:407] Connection header received
0926 13:44:07.221327 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:07.221333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:07.221557 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:07.221973 (+ 416us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:07.222083 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.223548 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.225445 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.226309 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.226467 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:07.226659 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:07.226712 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 13:44:07.226808 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":73,"threads_started":1}
I20260926 13:44:07.228322 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.220250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:39321 (local address 127.0.0.1:52066)
0926 13:44:07.221218 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:07.221233 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:07.221315 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:07.221904 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:07.221908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:07.221928 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:07.222025 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:07.222031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.223674 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.223677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:07.225283 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:07.225295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.227857 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.227861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:07.227973 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:07.227997 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 13:44:07.228082 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:44:07.228157 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":163,"thread_start_us":130,"threads_started":1}
I20260926 13:44:07.228452 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.220744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52066 (local address 127.1.49.3:39321)
0926 13:44:07.221307 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:07.221325 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:07.221350 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:44:07.221427 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:07.221432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:07.221602 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:07.221748 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:07.222181 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.223500 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.225459 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.227432 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.227571 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:07.227758 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:07.228136 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 13:44:07.228254 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":161,"threads_started":1}
I20260926 13:44:07.229557 19033 tablet_service.cc:1467] Processing CreateTablet for tablet f37284672391447d9ca3b2a0f6a415fd (DEFAULT_TABLE table=test-table [id=833a16baff4a41c890f339cddcc759e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:07.230048 18902 tablet_service.cc:1467] Processing CreateTablet for tablet f37284672391447d9ca3b2a0f6a415fd (DEFAULT_TABLE table=test-table [id=833a16baff4a41c890f339cddcc759e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:07.230756 19033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37284672391447d9ca3b2a0f6a415fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:07.231138 18902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37284672391447d9ca3b2a0f6a415fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:07.234927 19164 tablet_service.cc:1467] Processing CreateTablet for tablet f37284672391447d9ca3b2a0f6a415fd (DEFAULT_TABLE table=test-table [id=833a16baff4a41c890f339cddcc759e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:07.236213 19164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37284672391447d9ca3b2a0f6a415fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:07.237936 19247 tablet_bootstrap.cc:492] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0: Bootstrap starting.
I20260926 13:44:07.239612 19248 tablet_bootstrap.cc:492] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403: Bootstrap starting.
I20260926 13:44:07.240417 19247 tablet_bootstrap.cc:654] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:07.241191 19247 log.cc:824] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:07.241987 19248 tablet_bootstrap.cc:654] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:07.243031 19248 log.cc:824] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:07.243414 19247 tablet_bootstrap.cc:492] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0: No bootstrap required, opened a new log
I20260926 13:44:07.243656 19247 ts_tablet_manager.cc:1397] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:44:07.244526 19251 tablet_bootstrap.cc:492] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098: Bootstrap starting.
I20260926 13:44:07.245635 19248 tablet_bootstrap.cc:492] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403: No bootstrap required, opened a new log
I20260926 13:44:07.245852 19248 ts_tablet_manager.cc:1397] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:44:07.246701 19251 tablet_bootstrap.cc:654] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:07.247440 19251 log.cc:824] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:07.248772 19247 raft_consensus.cc:348] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37445 } } peers { permanent_uuid: "6fd6a798769841cca8b46ac22dfd75f0" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 40019 } } peers { permanent_uuid: "3958833f832544eab2dccb2a488e1098" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39321 } }
I20260926 13:44:07.249212 19247 raft_consensus.cc:374] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:07.249321 19247 raft_consensus.cc:729] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd6a798769841cca8b46ac22dfd75f0, State: Initialized, Role: FOLLOWER
I20260926 13:44:07.249845 19247 consensus_queue.cc:260] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37445 } } peers { permanent_uuid: "6fd6a798769841cca8b46ac22dfd75f0" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 40019 } } peers { permanent_uuid: "3958833f832544eab2dccb2a488e1098" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39321 } }
I20260926 13:44:07.250270 19251 tablet_bootstrap.cc:492] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098: No bootstrap required, opened a new log
I20260926 13:44:07.250536 19251 ts_tablet_manager.cc:1397] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:44:07.250700 18968 heartbeater.cc:503] Master 127.1.49.62:36381 was elected leader, sending a full tablet report...
I20260926 13:44:07.251025 19247 ts_tablet_manager.cc:1428] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 13:44:07.250921 19248 raft_consensus.cc:348] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37445 } } peers { permanent_uuid: "6fd6a798769841cca8b46ac22dfd75f0" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 40019 } } peers { permanent_uuid: "3958833f832544eab2dccb2a488e1098" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39321 } }
I20260926 13:44:07.251245 19248 raft_consensus.cc:374] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:07.251315 19248 raft_consensus.cc:729] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb43a5ac7824ebc8b62a07d4d9a7403, State: Initialized, Role: FOLLOWER
I20260926 13:44:07.251791 19248 consensus_queue.cc:260] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37445 } } peers { permanent_uuid: "6fd6a798769841cca8b46ac22dfd75f0" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 40019 } } peers { permanent_uuid: "3958833f832544eab2dccb2a488e1098" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39321 } }
W20260926 13:44:07.252466 18969 tablet.cc:2369] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:44:07.252929 19248 ts_tablet_manager.cc:1428] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 13:44:07.253291 19099 heartbeater.cc:503] Master 127.1.49.62:36381 was elected leader, sending a full tablet report...
I20260926 13:44:07.255592 19251 raft_consensus.cc:348] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37445 } } peers { permanent_uuid: "6fd6a798769841cca8b46ac22dfd75f0" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 40019 } } peers { permanent_uuid: "3958833f832544eab2dccb2a488e1098" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39321 } }
I20260926 13:44:07.256006 19251 raft_consensus.cc:374] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:07.256104 19251 raft_consensus.cc:729] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3958833f832544eab2dccb2a488e1098, State: Initialized, Role: FOLLOWER
I20260926 13:44:07.256785 19251 consensus_queue.cc:260] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37445 } } peers { permanent_uuid: "6fd6a798769841cca8b46ac22dfd75f0" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 40019 } } peers { permanent_uuid: "3958833f832544eab2dccb2a488e1098" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39321 } }
I20260926 13:44:07.257570 19230 heartbeater.cc:503] Master 127.1.49.62:36381 was elected leader, sending a full tablet report...
I20260926 13:44:07.257845 19251 ts_tablet_manager.cc:1428] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 13:44:07.299926 19253 raft_consensus.cc:482] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:44:07.300180 19253 raft_consensus.cc:504] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37445 } } peers { permanent_uuid: "6fd6a798769841cca8b46ac22dfd75f0" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 40019 } } peers { permanent_uuid: "3958833f832544eab2dccb2a488e1098" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39321 } }
I20260926 13:44:07.301471 19253 leader_election.cc:288] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccb43a5ac7824ebc8b62a07d4d9a7403 (127.1.49.2:37445), 3958833f832544eab2dccb2a488e1098 (127.1.49.3:39321)
I20260926 13:44:07.306360 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.302144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:54371 (local address 127.1.49.2:37445)
0926 13:44:07.302299 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:07.302303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:07.302320 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:44:07.302399 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:07.302405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:07.302468 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:07.302563 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:07.302894 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.303533 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.304897 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.305488 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.305634 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:07.305708 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:07.306148 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 13:44:07.306224 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 13:44:07.306360 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.302148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:58849 (local address 127.1.49.3:39321)
0926 13:44:07.302286 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:07.302289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:07.302437 (+ 148us) server_negotiation.cc:407] Connection header received
0926 13:44:07.302586 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:07.302589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:07.302636 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:07.302716 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:07.303089 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.303630 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.305073 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.305641 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.305724 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:07.305773 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:07.306133 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 13:44:07.306224 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 13:44:07.306424 19256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.301675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:37445 (local address 127.1.49.1:54371)
0926 13:44:07.301971 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:07.301987 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:07.302099 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:07.302586 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:07.302590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:07.302610 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:07.302757 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:07.302764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.303702 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.303708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:07.304763 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:07.304796 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.305772 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.305775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:07.305897 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:07.305990 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 13:44:07.306078 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 13:44:07.306212 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":101,"thread_start_us":142,"threads_started":1}
I20260926 13:44:07.306428 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.301885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:39321 (local address 127.1.49.1:58849)
0926 13:44:07.302345 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:07.302361 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:07.302477 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:07.302748 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:07.302751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:07.302765 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:07.302933 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:07.302941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.303753 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.303755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:07.304936 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:07.304947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.305743 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.305747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:07.305867 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:07.305989 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 13:44:07.306078 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 13:44:07.306213 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":71,"thread_start_us":146,"threads_started":1}
I20260926 13:44:07.307349 19184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37284672391447d9ca3b2a0f6a415fd" candidate_uuid: "6fd6a798769841cca8b46ac22dfd75f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3958833f832544eab2dccb2a488e1098" is_pre_election: true
I20260926 13:44:07.307349 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37284672391447d9ca3b2a0f6a415fd" candidate_uuid: "6fd6a798769841cca8b46ac22dfd75f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403" is_pre_election: true
I20260926 13:44:07.307703 19184 raft_consensus.cc:2454] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fd6a798769841cca8b46ac22dfd75f0 in term 0.
I20260926 13:44:07.307730 19053 raft_consensus.cc:2454] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fd6a798769841cca8b46ac22dfd75f0 in term 0.
I20260926 13:44:07.308446 18858 leader_election.cc:302] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3958833f832544eab2dccb2a488e1098, 6fd6a798769841cca8b46ac22dfd75f0, ccb43a5ac7824ebc8b62a07d4d9a7403; no voters:
I20260926 13:44:07.308841 19253 raft_consensus.cc:2781] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:44:07.308959 19253 raft_consensus.cc:482] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:44:07.309026 19253 raft_consensus.cc:3037] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:07.310194 19253 raft_consensus.cc:504] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37445 } } peers { permanent_uuid: "6fd6a798769841cca8b46ac22dfd75f0" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 40019 } } peers { permanent_uuid: "3958833f832544eab2dccb2a488e1098" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39321 } }
I20260926 13:44:07.310631 19253 leader_election.cc:288] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [CANDIDATE]: Term 1 election: Requested vote from peers ccb43a5ac7824ebc8b62a07d4d9a7403 (127.1.49.2:37445), 3958833f832544eab2dccb2a488e1098 (127.1.49.3:39321)
I20260926 13:44:07.311141 19053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37284672391447d9ca3b2a0f6a415fd" candidate_uuid: "6fd6a798769841cca8b46ac22dfd75f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403"
I20260926 13:44:07.311154 19184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37284672391447d9ca3b2a0f6a415fd" candidate_uuid: "6fd6a798769841cca8b46ac22dfd75f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3958833f832544eab2dccb2a488e1098"
I20260926 13:44:07.311293 19053 raft_consensus.cc:3037] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:07.311293 19184 raft_consensus.cc:3037] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:07.312712 19184 raft_consensus.cc:2454] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fd6a798769841cca8b46ac22dfd75f0 in term 1.
I20260926 13:44:07.312712 19053 raft_consensus.cc:2454] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fd6a798769841cca8b46ac22dfd75f0 in term 1.
I20260926 13:44:07.313126 18855 leader_election.cc:302] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3958833f832544eab2dccb2a488e1098, 6fd6a798769841cca8b46ac22dfd75f0, ccb43a5ac7824ebc8b62a07d4d9a7403; no voters:
I20260926 13:44:07.313397 19253 raft_consensus.cc:2781] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:07.313755 19253 raft_consensus.cc:686] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6fd6a798769841cca8b46ac22dfd75f0, State: Running, Role: LEADER
I20260926 13:44:07.314195 19253 consensus_queue.cc:237] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37445 } } peers { permanent_uuid: "6fd6a798769841cca8b46ac22dfd75f0" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 40019 } } peers { permanent_uuid: "3958833f832544eab2dccb2a488e1098" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39321 } }
I20260926 13:44:07.318324 18780 catalog_manager.cc:5495] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fd6a798769841cca8b46ac22dfd75f0 (127.1.49.1). New cstate: current_term: 1 leader_uuid: "6fd6a798769841cca8b46ac22dfd75f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fd6a798769841cca8b46ac22dfd75f0" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 40019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3958833f832544eab2dccb2a488e1098" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39321 } health_report { overall_health: UNKNOWN } } }
I20260926 13:44:07.334714 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.330497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.1.49.1:40019)
0926 13:44:07.330665 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:07.330672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:07.330690 (+ 18us) server_negotiation.cc:407] Connection header received
0926 13:44:07.330796 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:07.330798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:07.330861 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:07.331015 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:07.331289 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.331967 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.333005 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:07.333311 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:07.333444 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:07.333802 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:07.334270 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:07.334416 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:07.334483 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 13:44:07.334544 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 13:44:07.341670 19053 raft_consensus.cc:1261] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403 [term 1 FOLLOWER]: Refusing update from remote peer 6fd6a798769841cca8b46ac22dfd75f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:07.342454 19184 raft_consensus.cc:1261] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098 [term 1 FOLLOWER]: Refusing update from remote peer 6fd6a798769841cca8b46ac22dfd75f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:07.342690 19253 consensus_queue.cc:1035] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb43a5ac7824ebc8b62a07d4d9a7403" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:07.343225 19258 consensus_queue.cc:1035] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3958833f832544eab2dccb2a488e1098" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:07.351344 19261 mvcc.cc:205] Tried to move back new op lower bound from 7333602293101604864 to 7333602293002014720. Current Snapshot: MvccSnapshot[applied={T|T < 7333602293101604864}]
I20260926 13:44:07.354424 19262 mvcc.cc:205] Tried to move back new op lower bound from 7333602293101604864 to 7333602293002014720. Current Snapshot: MvccSnapshot[applied={T|T < 7333602293101604864}]
I20260926 13:44:07.360962 18780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:07.361126 18780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:07.363849 18780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d60fe5e0bb04d35a97c6bc18f7a3341: Sending DeleteTablet for 3 replicas of tablet f37284672391447d9ca3b2a0f6a415fd
I20260926 13:44:07.364774 18902 tablet_service.cc:1514] Processing DeleteTablet for tablet f37284672391447d9ca3b2a0f6a415fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:07 UTC) from {username='slave'} at 127.0.0.1:58072
I20260926 13:44:07.364784 19033 tablet_service.cc:1514] Processing DeleteTablet for tablet f37284672391447d9ca3b2a0f6a415fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:07 UTC) from {username='slave'} at 127.0.0.1:53032
I20260926 13:44:07.364979 19164 tablet_service.cc:1514] Processing DeleteTablet for tablet f37284672391447d9ca3b2a0f6a415fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:07 UTC) from {username='slave'} at 127.0.0.1:52066
I20260926 13:44:07.365351 19271 tablet_replica.cc:330] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0: stopping tablet replica
I20260926 13:44:07.365738 19271 raft_consensus.cc:2229] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:44:07.366063 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 18840
I20260926 13:44:07.366150 19271 raft_consensus.cc:2258] T f37284672391447d9ca3b2a0f6a415fd P 6fd6a798769841cca8b46ac22dfd75f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:07.368041 19272 tablet_replica.cc:330] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403: stopping tablet replica
I20260926 13:44:07.368418 19272 raft_consensus.cc:2229] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:44:07.368753 19272 raft_consensus.cc:2258] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:07.369182 19273 tablet_replica.cc:330] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098: stopping tablet replica
I20260926 13:44:07.369449 19273 raft_consensus.cc:2229] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:44:07.369724 19273 raft_consensus.cc:2258] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:07.371241 19273 ts_tablet_manager.cc:1905] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:07.371425 19272 ts_tablet_manager.cc:1905] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:07.373944 19273 ts_tablet_manager.cc:1918] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:44:07.374044 19273 log.cc:1196] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/f37284672391447d9ca3b2a0f6a415fd
W20260926 13:44:07.374220 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:40019 (error 108)
I20260926 13:44:07.374354 19273 ts_tablet_manager.cc:1939] T f37284672391447d9ca3b2a0f6a415fd P 3958833f832544eab2dccb2a488e1098: Deleting consensus metadata
I20260926 13:44:07.374913 19272 ts_tablet_manager.cc:1918] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:44:07.375020 19272 log.cc:1196] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/f37284672391447d9ca3b2a0f6a415fd
I20260926 13:44:07.375033 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 18971
I20260926 13:44:07.375358 19272 ts_tablet_manager.cc:1939] T f37284672391447d9ca3b2a0f6a415fd P ccb43a5ac7824ebc8b62a07d4d9a7403: Deleting consensus metadata
I20260926 13:44:07.375785 18767 catalog_manager.cc:4841] TS 3958833f832544eab2dccb2a488e1098 (127.1.49.3:39321): tablet f37284672391447d9ca3b2a0f6a415fd (table test-table [id=833a16baff4a41c890f339cddcc759e7]) successfully deleted
I20260926 13:44:07.375787 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.375304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:40019 (local address 127.0.0.1:58102)
0926 13:44:07.375568 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:07.375658 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:40019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260926 13:44:07.376051 18766 catalog_manager.cc:4584] TS 6fd6a798769841cca8b46ac22dfd75f0 (127.1.49.1:40019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f37284672391447d9ca3b2a0f6a415fd: Network error: Client connection negotiation failed: client connection to 127.1.49.1:40019: connect: Connection refused (error 111)
I20260926 13:44:07.381686 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 19102
I20260926 13:44:07.383391 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:07.381905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:37445 (local address 127.0.0.1:53038)
0926 13:44:07.383184 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:07.383264 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.2:37445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1158}
I20260926 13:44:07.388003 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 18749
2026-09-26T13:44:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1125 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:07 dist-test-slave-1wns krb5kdc[19277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:07 dist-test-slave-1wns krb5kdc[19277](info): set up 2 sockets
Sep 26 13:44:07 dist-test-slave-1wns krb5kdc[19277](info): commencing operation
krb5kdc: starting...
W20260926 13:44:08.476531 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:08 dist-test-slave-1wns krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:44:08.613509 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:41379
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:41815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:41379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:08.729549 19290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:08.729916 19290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:08.730010 19290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:08.734225 19290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:08.734341 19290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:08.734378 19290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:08.734421 19290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:08.734478 19290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:08.740836 19290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:41815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:41379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:41379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:08.742614 19290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:08.743745 19290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:08.753341 19295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:08.753360 19298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:08.753350 19296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:08.754019 19290 server_base.cc:1034] running on GCE node
I20260926 13:44:08.754638 19290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:08.755946 19290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:08.757179 19290 hybrid_clock.cc:648] HybridClock initialized: now 1790430248757160 us; error 37 us; skew 500 ppm
Sep 26 13:44:08 dist-test-slave-1wns krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430248, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:08.765720 19290 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:44:08.766809 19290 webserver.cc:460] Webserver started at http://127.1.49.62:45919/ using document root <none> and password file <none>
I20260926 13:44:08.767441 19290 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:08.767540 19290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:08.767796 19290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:08.770126 19290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "f78e817fc68143728404eb75e27582f8"
format_stamp: "Formatted at 2026-09-26 13:44:08 on dist-test-slave-1wns"
server_key: "190a0e39dd4823087ccd4343dc18f196"
server_key_iv: "2f0833f25316d95a5230bb0b1f8fecac"
server_key_version: "encryptionkey@0"
I20260926 13:44:08.770761 19290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "f78e817fc68143728404eb75e27582f8"
format_stamp: "Formatted at 2026-09-26 13:44:08 on dist-test-slave-1wns"
server_key: "190a0e39dd4823087ccd4343dc18f196"
server_key_iv: "2f0833f25316d95a5230bb0b1f8fecac"
server_key_version: "encryptionkey@0"
I20260926 13:44:08.774997 19290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 13:44:08.777729 19305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:08.779181 19290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:44:08.779345 19290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "f78e817fc68143728404eb75e27582f8"
format_stamp: "Formatted at 2026-09-26 13:44:08 on dist-test-slave-1wns"
server_key: "190a0e39dd4823087ccd4343dc18f196"
server_key_iv: "2f0833f25316d95a5230bb0b1f8fecac"
server_key_version: "encryptionkey@0"
I20260926 13:44:08.779484 19290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:08.809661 19290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:08.813546 19290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:08.813853 19290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:08.822031 19290 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:41379
I20260926 13:44:08.822039 19357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:41379 every 8 connection(s)
I20260926 13:44:08.823099 19290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:08.826282 19358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:08.832017 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 19290
I20260926 13:44:08.832168 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:08.832495 1220 external_mini_cluster.cc:1428] Setting key 33202413f762092256e76969f632dbbc
I20260926 13:44:08.833097 19358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8: Bootstrap starting.
I20260926 13:44:08.835966 19358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:08.836771 19358 log.cc:824] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:08.838786 19358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8: No bootstrap required, opened a new log
Sep 26 13:44:08 dist-test-slave-1wns krb5kdc[19277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:08.842819 19358 raft_consensus.cc:348] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78e817fc68143728404eb75e27582f8" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41379 } }
I20260926 13:44:08.843168 19358 raft_consensus.cc:374] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:08.843235 19358 raft_consensus.cc:729] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78e817fc68143728404eb75e27582f8, State: Initialized, Role: FOLLOWER
I20260926 13:44:08.843734 19358 consensus_queue.cc:260] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78e817fc68143728404eb75e27582f8" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41379 } }
I20260926 13:44:08.843923 19358 raft_consensus.cc:388] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:08.844009 19358 raft_consensus.cc:482] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:08.844118 19358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:08.845351 19358 raft_consensus.cc:504] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78e817fc68143728404eb75e27582f8" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41379 } }
I20260926 13:44:08.845736 19358 leader_election.cc:302] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f78e817fc68143728404eb75e27582f8; no voters:
I20260926 13:44:08.846097 19358 leader_election.cc:288] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:08.846182 19363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:08.846411 19363 raft_consensus.cc:686] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [term 1 LEADER]: Becoming Leader. State: Replica: f78e817fc68143728404eb75e27582f8, State: Running, Role: LEADER
I20260926 13:44:08.846733 19363 consensus_queue.cc:237] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78e817fc68143728404eb75e27582f8" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41379 } }
I20260926 13:44:08.847543 19358 sys_catalog.cc:564] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:08.848008 19363 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f78e817fc68143728404eb75e27582f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78e817fc68143728404eb75e27582f8" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41379 } } }
I20260926 13:44:08.848129 19363 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:08.848526 19367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:08.850042 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:08.834101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.1.49.62:41379)
0926 13:44:08.834725 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:08.834747 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:08.834796 (+ 49us) server_negotiation.cc:407] Connection header received
0926 13:44:08.835597 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:08.835620 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:08.836189 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:08.836628 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:08.837685 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:08.838800 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:08.839875 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:08.840435 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:08.840644 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:08.843792 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:08.843797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:08.843876 (+ 79us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:08.843933 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:08.846023 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:08.846599 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:08.846617 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:08.846626 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:08.846818 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:08.847153 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:08.847156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:08.847158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:08.847684 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:08.847902 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:08.848548 (+ 646us) server_negotiation.cc:299] Negotiation successful
0926 13:44:08.848904 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":196,"threads_started":1}
I20260926 13:44:08.851866 19367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:08.851825 19364 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f78e817fc68143728404eb75e27582f8. Latest consensus state: current_term: 1 leader_uuid: "f78e817fc68143728404eb75e27582f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78e817fc68143728404eb75e27582f8" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41379 } } }
I20260926 13:44:08.851930 19364 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:08.858506 19367 catalog_manager.cc:1347] Generated new cluster ID: 1c72dfbf39044e30b67889bf00b05fc1
I20260926 13:44:08.858594 19367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:08.891876 19367 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:08.893324 19367 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:08.900687 19367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8: Generated new TSK 0
I20260926 13:44:08.901611 19367 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:44:08.944536 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41379
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:41815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:09.067509 19384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:09.067852 19384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:09.067900 19384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:09.072220 19384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:09.072295 19384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:09.072358 19384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:44:09.077965 19384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:41815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:09.079064 19384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:09.080103 19384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:09.090608 19390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:09.090631 19389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:09.090636 19392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:09.090904 19384 server_base.cc:1034] running on GCE node
I20260926 13:44:09.091504 19384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:09.092329 19384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:09.093528 19384 hybrid_clock.cc:648] HybridClock initialized: now 1790430249093470 us; error 70 us; skew 500 ppm
Sep 26 13:44:09 dist-test-slave-1wns krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:09.097318 19384 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:44:09.098519 19384 webserver.cc:460] Webserver started at http://127.1.49.1:40529/ using document root <none> and password file <none>
I20260926 13:44:09.099210 19384 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:09.099326 19384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:09.099617 19384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:09.102074 19384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "b09eddbf874f4fd4a852be7d42f4216e"
format_stamp: "Formatted at 2026-09-26 13:44:09 on dist-test-slave-1wns"
server_key: "731eebaac0c38cc5f0a644b825a40d40"
server_key_iv: "8b018e788c8f7140b771424b1c54f243"
server_key_version: "encryptionkey@0"
I20260926 13:44:09.102833 19384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "b09eddbf874f4fd4a852be7d42f4216e"
format_stamp: "Formatted at 2026-09-26 13:44:09 on dist-test-slave-1wns"
server_key: "731eebaac0c38cc5f0a644b825a40d40"
server_key_iv: "8b018e788c8f7140b771424b1c54f243"
server_key_version: "encryptionkey@0"
I20260926 13:44:09.107432 19384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 13:44:09.110528 19399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:09.112191 19384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:09.112346 19384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "b09eddbf874f4fd4a852be7d42f4216e"
format_stamp: "Formatted at 2026-09-26 13:44:09 on dist-test-slave-1wns"
server_key: "731eebaac0c38cc5f0a644b825a40d40"
server_key_iv: "8b018e788c8f7140b771424b1c54f243"
server_key_version: "encryptionkey@0"
I20260926 13:44:09.112481 19384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:09.137902 19384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:09.141484 19384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:09.141706 19384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:09.142488 19384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:09.143373 19384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:09.143445 19384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:09.143515 19384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:09.143558 19384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:09.153301 19384 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:34257
I20260926 13:44:09.153324 19512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:34257 every 8 connection(s)
I20260926 13:44:09.154232 19384 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:44:09.162495 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 19384
I20260926 13:44:09.162635 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:44:09.162989 1220 external_mini_cluster.cc:1428] Setting key 5934c180eae9a6efda8c6e920f8e276a
Sep 26 13:44:09 dist-test-slave-1wns krb5kdc[19277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:09.170459 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.156098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:41271 (local address 127.1.49.62:41379)
0926 13:44:09.156230 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:09.156234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:09.157517 (+ 1283us) server_negotiation.cc:407] Connection header received
0926 13:44:09.158570 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:09.158573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:09.158632 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:09.158713 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:09.160132 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.160664 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.162335 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.162748 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.162841 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.165602 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:09.165610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:09.165617 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:09.165655 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:09.167637 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:09.168468 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:09.168471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:09.168472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:09.168560 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:09.169049 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:09.169052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:09.169053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:09.169259 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:09.169342 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:09.170182 (+ 840us) server_negotiation.cc:299] Negotiation successful
0926 13:44:09.170321 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:44:09.172034 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.156503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41379 (local address 127.1.49.1:41271)
0926 13:44:09.157354 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:09.157402 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:09.158339 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:09.158866 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:09.158873 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:09.159329 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:09.159961 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.159979 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.160821 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.160824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.162193 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.162209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.162917 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.162922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.163127 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.163861 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:09.163882 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:09.165417 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:09.167788 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:09.167830 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:09.167853 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:09.168307 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:09.168668 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:09.168675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:09.168681 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:09.168918 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:09.169392 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:09.169402 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:09.169858 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 13:44:09.170119 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 13:44:09.170574 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":118,"threads_started":1}
I20260926 13:44:09.173476 19513 heartbeater.cc:347] Connected to a master server at 127.1.49.62:41379
I20260926 13:44:09.173758 19513 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:09.174293 19513 heartbeater.cc:511] Master 127.1.49.62:41379 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
I20260926 13:44:09.175918 19322 ts_manager.cc:194] Registered new tserver with Master: b09eddbf874f4fd4a852be7d42f4216e (127.1.49.1:34257)
I20260926 13:44:09.177089 19322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:41271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:44:09.195304 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41379
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:41815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:09.316007 19518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:09.316357 19518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:09.316439 19518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:09.321058 19518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:09.321165 19518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:09.321305 19518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:44:09.327379 19518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:41815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:09.328689 19518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:09.329815 19518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:09.340492 19523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:09.340548 19524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:09.340559 19518 server_base.cc:1034] running on GCE node
W20260926 13:44:09.340492 19526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:09.341192 19518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:09.341939 19518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:09.343118 19518 hybrid_clock.cc:648] HybridClock initialized: now 1790430249343079 us; error 51 us; skew 500 ppm
Sep 26 13:44:09 dist-test-slave-1wns krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:09.346294 19518 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:44:09.347254 19518 webserver.cc:460] Webserver started at http://127.1.49.2:39851/ using document root <none> and password file <none>
I20260926 13:44:09.348075 19518 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:09.348174 19518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:09.348416 19518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:09.350844 19518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "df8132b7c0b244479c4b7a6d431c1219"
format_stamp: "Formatted at 2026-09-26 13:44:09 on dist-test-slave-1wns"
server_key: "956ceede577b365858dfcbe975a6bf9a"
server_key_iv: "592de570eb6246fa8ad29dcc37f2d6df"
server_key_version: "encryptionkey@0"
I20260926 13:44:09.351522 19518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "df8132b7c0b244479c4b7a6d431c1219"
format_stamp: "Formatted at 2026-09-26 13:44:09 on dist-test-slave-1wns"
server_key: "956ceede577b365858dfcbe975a6bf9a"
server_key_iv: "592de570eb6246fa8ad29dcc37f2d6df"
server_key_version: "encryptionkey@0"
I20260926 13:44:09.355661 19518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 13:44:09.358757 19533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:09.360167 19518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 13:44:09.360319 19518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "df8132b7c0b244479c4b7a6d431c1219"
format_stamp: "Formatted at 2026-09-26 13:44:09 on dist-test-slave-1wns"
server_key: "956ceede577b365858dfcbe975a6bf9a"
server_key_iv: "592de570eb6246fa8ad29dcc37f2d6df"
server_key_version: "encryptionkey@0"
I20260926 13:44:09.360447 19518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:09.389557 19518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:09.393539 19518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:09.393790 19518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:09.394465 19518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:09.395407 19518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:09.395481 19518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:09.395560 19518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:09.395605 19518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:09.406248 19518 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:35721
I20260926 13:44:09.406315 19646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:35721 every 8 connection(s)
I20260926 13:44:09.407282 19518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:44:09.412719 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 19518
I20260926 13:44:09.412822 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:44:09.413098 1220 external_mini_cluster.cc:1428] Setting key bf46c4f47d511c7272f5e1c35f8c95b0
Sep 26 13:44:09 dist-test-slave-1wns krb5kdc[19277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
I20260926 13:44:09.422348 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.409471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:51019 (local address 127.1.49.62:41379)
0926 13:44:09.409664 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:09.409668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:09.410885 (+ 1217us) server_negotiation.cc:407] Connection header received
0926 13:44:09.411860 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:09.411864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:09.411925 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:09.412034 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:09.413423 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.414073 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.415395 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.415629 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.415774 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.418524 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:09.418529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:09.418532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:09.418567 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:09.420160 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:09.420721 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:09.420724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:09.420725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:09.420803 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:09.421161 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:09.421164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:09.421166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:09.421361 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:09.421470 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:09.422045 (+ 575us) server_negotiation.cc:299] Negotiation successful
0926 13:44:09.422177 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 13:44:09.423233 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.409948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41379 (local address 127.1.49.2:51019)
0926 13:44:09.410695 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:09.410758 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:09.411615 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:09.412222 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:09.412229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:09.412640 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:09.413261 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.413291 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.414215 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.414224 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.415243 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.415253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.415878 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.415883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.416059 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.416752 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:09.416770 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:09.418334 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:09.420288 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:09.420295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:09.420313 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:09.420588 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:09.420898 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:09.420902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:09.420904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:09.421055 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:09.421469 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:09.421477 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:09.421787 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 13:44:09.421995 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 13:44:09.422364 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":136,"threads_started":1}
I20260926 13:44:09.424834 19647 heartbeater.cc:347] Connected to a master server at 127.1.49.62:41379
I20260926 13:44:09.425096 19647 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
I20260926 13:44:09.425720 19647 heartbeater.cc:511] Master 127.1.49.62:41379 requested a full tablet report, sending...
I20260926 13:44:09.426980 19322 ts_manager.cc:194] Registered new tserver with Master: df8132b7c0b244479c4b7a6d431c1219 (127.1.49.2:35721)
I20260926 13:44:09.427974 19322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:51019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:44:09.444216 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41379
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:41815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:09.563321 19652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:09.563637 19652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:09.563699 19652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:09.568409 19652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:09.568490 19652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:09.568555 19652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:44:09.574154 19652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:41815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:41379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:09.575275 19652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:09.576328 19652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:09.586611 19658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:09.586611 19657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:09.586810 19652 server_base.cc:1034] running on GCE node
W20260926 13:44:09.586637 19660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:09.587493 19652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:09.588313 19652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:09.589526 19652 hybrid_clock.cc:648] HybridClock initialized: now 1790430249589509 us; error 32 us; skew 500 ppm
Sep 26 13:44:09 dist-test-slave-1wns krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:09.593183 19652 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:44:09.594219 19652 webserver.cc:460] Webserver started at http://127.1.49.3:43249/ using document root <none> and password file <none>
I20260926 13:44:09.594852 19652 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:09.594921 19652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:09.595163 19652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:09.597450 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "43afb6c03bce4a509c28cc52a0db2d3c"
format_stamp: "Formatted at 2026-09-26 13:44:09 on dist-test-slave-1wns"
server_key: "3048d1b206e23acaca7bed2084a5faa6"
server_key_iv: "86b322416ee0b962af51fbfe4b8f66e3"
server_key_version: "encryptionkey@0"
I20260926 13:44:09.598055 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "43afb6c03bce4a509c28cc52a0db2d3c"
format_stamp: "Formatted at 2026-09-26 13:44:09 on dist-test-slave-1wns"
server_key: "3048d1b206e23acaca7bed2084a5faa6"
server_key_iv: "86b322416ee0b962af51fbfe4b8f66e3"
server_key_version: "encryptionkey@0"
I20260926 13:44:09.602450 19652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:44:09.605365 19667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:09.606841 19652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:44:09.606983 19652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "43afb6c03bce4a509c28cc52a0db2d3c"
format_stamp: "Formatted at 2026-09-26 13:44:09 on dist-test-slave-1wns"
server_key: "3048d1b206e23acaca7bed2084a5faa6"
server_key_iv: "86b322416ee0b962af51fbfe4b8f66e3"
server_key_version: "encryptionkey@0"
I20260926 13:44:09.607113 19652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:09.636018 19652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:09.639945 19652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:09.640213 19652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:09.641043 19652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:09.641943 19652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:09.641994 19652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:09.642041 19652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:09.642058 19652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:09.651592 19652 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:46299
I20260926 13:44:09.651669 19780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:46299 every 8 connection(s)
I20260926 13:44:09.652590 19652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:44:09.662418 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 19652
I20260926 13:44:09.662592 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:44:09.662940 1220 external_mini_cluster.cc:1428] Setting key 1a62fb982cc810e0e051c70aae8fd08c
Sep 26 13:44:09 dist-test-slave-1wns krb5kdc[19277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:09.668147 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.654726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:53497 (local address 127.1.49.62:41379)
0926 13:44:09.654970 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:09.654974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:09.656227 (+ 1253us) server_negotiation.cc:407] Connection header received
0926 13:44:09.657260 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:09.657264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:09.657324 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:09.657445 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:09.658908 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.659502 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.660888 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.661142 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.661268 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.664311 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:09.664316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:09.664319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:09.664350 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:09.665948 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:09.666521 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:09.666529 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:09.666531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:09.666609 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:09.666976 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:09.666979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:09.666981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:09.667182 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:09.667291 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:09.667857 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 13:44:09.667986 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 13:44:09.669018 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.655135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41379 (local address 127.1.49.3:53497)
0926 13:44:09.656056 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:09.656109 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:09.657031 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:09.657630 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:09.657638 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:09.658146 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:09.658746 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.658762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.659665 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.659668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.660749 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.660759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.661319 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.661324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.661506 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.662130 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:09.662148 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:09.664084 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:09.666079 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:09.666086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:09.666103 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:09.666401 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:09.666709 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:09.666713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:09.666715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:09.666871 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:09.667288 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:09.667296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:09.667564 (+ 268us) client_negotiation.cc:769] Sending connection context
0926 13:44:09.667773 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 13:44:09.668148 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":119,"threads_started":1}
I20260926 13:44:09.670308 19781 heartbeater.cc:347] Connected to a master server at 127.1.49.62:41379
I20260926 13:44:09.670581 19781 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:09.671182 19781 heartbeater.cc:511] Master 127.1.49.62:41379 requested a full tablet report, sending...
I20260926 13:44:09.672607 19322 ts_manager.cc:194] Registered new tserver with Master: 43afb6c03bce4a509c28cc52a0db2d3c (127.1.49.3:46299)
I20260926 13:44:09.673252 19322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:53497
I20260926 13:44:09.677531 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:44:09.688896 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.679871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48676 (local address 127.1.49.62:41379)
0926 13:44:09.680001 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:09.680005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:09.680252 (+ 247us) server_negotiation.cc:407] Connection header received
0926 13:44:09.680329 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:09.680332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:09.680377 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:09.680459 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:09.681357 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.681999 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.683120 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.683358 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.683451 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.685328 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:09.685331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:09.685333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:09.685360 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:09.686900 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:09.687474 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:09.687482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:09.687483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:09.687559 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:09.687929 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:09.687933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:09.687935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:09.688154 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:09.688316 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:09.688627 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 13:44:09.688725 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:44:09.692389 19322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:44:09.695556 19322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:44:09.712992 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.706626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58126 (local address 127.1.49.1:34257)
0926 13:44:09.707760 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:09.707768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:09.707789 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:44:09.707886 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:09.707891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:09.708097 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:09.708270 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:09.708544 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.709656 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.711415 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.712224 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.712362 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.712557 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:09.712694 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 13:44:09.712806 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1008,"thread_start_us":87,"threads_started":1}
I20260926 13:44:09.712983 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.705801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:34257 (local address 127.0.0.1:58126)
0926 13:44:09.706678 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:09.706737 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:09.706888 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:09.708245 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:09.708248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:09.708266 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:09.708400 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.708407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.709755 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.709761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.711226 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.711237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.712362 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.712366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.712490 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.712531 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 13:44:09.712664 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 13:44:09.712812 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":7,"spinlock_wait_cycles":4608,"thread_start_us":102,"threads_started":1}
I20260926 13:44:09.713230 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.706091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:35721 (local address 127.0.0.1:51164)
0926 13:44:09.707595 (+ 1504us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:09.707608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:09.707714 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:09.708472 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:09.708477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:09.708496 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:09.708581 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.708586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.709847 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.709851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.711260 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.711268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.712709 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.712713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.712808 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.712828 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 13:44:09.712905 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 13:44:09.713081 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1442,"mutex_wait_us":243,"thread_start_us":90,"threads_started":1}
I20260926 13:44:09.713608 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.705718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:46299 (local address 127.0.0.1:44524)
0926 13:44:09.706689 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:09.706737 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:09.706888 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:09.707758 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:09.707765 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:09.707795 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:09.707991 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.707999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.709553 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.709558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.711256 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.711268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.712650 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.712653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.712762 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.712828 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 13:44:09.712905 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 13:44:09.713005 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":177,"spinlock_wait_cycles":8320,"thread_start_us":121,"threads_started":1}
I20260926 13:44:09.713670 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.706469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.1.49.3:46299)
0926 13:44:09.707190 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:09.707199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:09.707219 (+ 20us) server_negotiation.cc:407] Connection header received
0926 13:44:09.707325 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:09.707331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:09.707570 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:09.707771 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:09.708129 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.709403 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.711408 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.712332 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.712443 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.712586 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:09.713419 (+ 833us) server_negotiation.cc:299] Negotiation successful
0926 13:44:09.713523 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":164,"threads_started":1}
I20260926 13:44:09.713430 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.707071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.1.49.2:35721)
0926 13:44:09.707441 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:09.707452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:09.707847 (+ 395us) server_negotiation.cc:407] Connection header received
0926 13:44:09.707934 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:09.707939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:09.708143 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:09.708303 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:09.708707 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.709451 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.711408 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.712332 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.712443 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.712581 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:09.712989 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 13:44:09.713159 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":151,"threads_started":1}
I20260926 13:44:09.717355 19447 tablet_service.cc:1467] Processing CreateTablet for tablet 45eff10609aa4fa1ab2c13bed419f485 (DEFAULT_TABLE table=test-table [id=425a173230b94cb1a787e44f07140de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:09.717360 19715 tablet_service.cc:1467] Processing CreateTablet for tablet 45eff10609aa4fa1ab2c13bed419f485 (DEFAULT_TABLE table=test-table [id=425a173230b94cb1a787e44f07140de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:09.717360 19581 tablet_service.cc:1467] Processing CreateTablet for tablet 45eff10609aa4fa1ab2c13bed419f485 (DEFAULT_TABLE table=test-table [id=425a173230b94cb1a787e44f07140de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:09.718576 19447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45eff10609aa4fa1ab2c13bed419f485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:09.718575 19715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45eff10609aa4fa1ab2c13bed419f485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:09.718575 19581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45eff10609aa4fa1ab2c13bed419f485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:09.725502 19798 tablet_bootstrap.cc:492] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e: Bootstrap starting.
I20260926 13:44:09.726486 19799 tablet_bootstrap.cc:492] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c: Bootstrap starting.
I20260926 13:44:09.727454 19800 tablet_bootstrap.cc:492] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219: Bootstrap starting.
I20260926 13:44:09.728341 19798 tablet_bootstrap.cc:654] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:09.728761 19799 tablet_bootstrap.cc:654] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:09.729746 19800 tablet_bootstrap.cc:654] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:09.730032 19799 log.cc:824] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:09.730062 19798 log.cc:824] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:09.731105 19800 log.cc:824] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:09.732355 19798 tablet_bootstrap.cc:492] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e: No bootstrap required, opened a new log
I20260926 13:44:09.732430 19799 tablet_bootstrap.cc:492] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c: No bootstrap required, opened a new log
I20260926 13:44:09.732591 19798 ts_tablet_manager.cc:1397] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 13:44:09.732635 19799 ts_tablet_manager.cc:1397] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:44:09.733211 19800 tablet_bootstrap.cc:492] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219: No bootstrap required, opened a new log
I20260926 13:44:09.733419 19800 ts_tablet_manager.cc:1397] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:44:09.736536 19798 raft_consensus.cc:348] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afb6c03bce4a509c28cc52a0db2d3c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 46299 } } peers { permanent_uuid: "b09eddbf874f4fd4a852be7d42f4216e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34257 } } peers { permanent_uuid: "df8132b7c0b244479c4b7a6d431c1219" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35721 } }
I20260926 13:44:09.737130 19798 raft_consensus.cc:374] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:09.737203 19798 raft_consensus.cc:729] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09eddbf874f4fd4a852be7d42f4216e, State: Initialized, Role: FOLLOWER
I20260926 13:44:09.737067 19799 raft_consensus.cc:348] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afb6c03bce4a509c28cc52a0db2d3c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 46299 } } peers { permanent_uuid: "b09eddbf874f4fd4a852be7d42f4216e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34257 } } peers { permanent_uuid: "df8132b7c0b244479c4b7a6d431c1219" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35721 } }
I20260926 13:44:09.737389 19799 raft_consensus.cc:374] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:09.737458 19799 raft_consensus.cc:729] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43afb6c03bce4a509c28cc52a0db2d3c, State: Initialized, Role: FOLLOWER
I20260926 13:44:09.737743 19798 consensus_queue.cc:260] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afb6c03bce4a509c28cc52a0db2d3c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 46299 } } peers { permanent_uuid: "b09eddbf874f4fd4a852be7d42f4216e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34257 } } peers { permanent_uuid: "df8132b7c0b244479c4b7a6d431c1219" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35721 } }
I20260926 13:44:09.737931 19799 consensus_queue.cc:260] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afb6c03bce4a509c28cc52a0db2d3c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 46299 } } peers { permanent_uuid: "b09eddbf874f4fd4a852be7d42f4216e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34257 } } peers { permanent_uuid: "df8132b7c0b244479c4b7a6d431c1219" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35721 } }
I20260926 13:44:09.737823 19800 raft_consensus.cc:348] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afb6c03bce4a509c28cc52a0db2d3c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 46299 } } peers { permanent_uuid: "b09eddbf874f4fd4a852be7d42f4216e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34257 } } peers { permanent_uuid: "df8132b7c0b244479c4b7a6d431c1219" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35721 } }
I20260926 13:44:09.738092 19800 raft_consensus.cc:374] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:09.738157 19800 raft_consensus.cc:729] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8132b7c0b244479c4b7a6d431c1219, State: Initialized, Role: FOLLOWER
I20260926 13:44:09.738611 19513 heartbeater.cc:503] Master 127.1.49.62:41379 was elected leader, sending a full tablet report...
I20260926 13:44:09.738782 19800 consensus_queue.cc:260] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afb6c03bce4a509c28cc52a0db2d3c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 46299 } } peers { permanent_uuid: "b09eddbf874f4fd4a852be7d42f4216e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34257 } } peers { permanent_uuid: "df8132b7c0b244479c4b7a6d431c1219" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35721 } }
I20260926 13:44:09.738950 19798 ts_tablet_manager.cc:1428] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:44:09.739032 19799 ts_tablet_manager.cc:1428] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:44:09.739428 19781 heartbeater.cc:503] Master 127.1.49.62:41379 was elected leader, sending a full tablet report...
I20260926 13:44:09.739737 19800 ts_tablet_manager.cc:1428] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 13:44:09.740121 19647 heartbeater.cc:503] Master 127.1.49.62:41379 was elected leader, sending a full tablet report...
I20260926 13:44:09.851742 19806 raft_consensus.cc:482] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:44:09.852038 19806 raft_consensus.cc:504] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afb6c03bce4a509c28cc52a0db2d3c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 46299 } } peers { permanent_uuid: "b09eddbf874f4fd4a852be7d42f4216e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34257 } } peers { permanent_uuid: "df8132b7c0b244479c4b7a6d431c1219" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35721 } }
I20260926 13:44:09.853439 19806 leader_election.cc:288] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43afb6c03bce4a509c28cc52a0db2d3c (127.1.49.3:46299), b09eddbf874f4fd4a852be7d42f4216e (127.1.49.1:34257)
I20260926 13:44:09.858173 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.853707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:46299 (local address 127.1.49.2:38745)
0926 13:44:09.853892 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:09.853919 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:09.854027 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:09.854424 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:09.854427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:09.854509 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:09.854667 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.854674 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.855678 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.855681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.856819 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.856830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.857605 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.857666 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.857784 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.857823 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 13:44:09.857910 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 13:44:09.857979 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":141,"spinlock_wait_cycles":3328}
I20260926 13:44:09.858245 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.853823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:38745 (local address 127.1.49.3:46299)
0926 13:44:09.853971 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:09.853976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:09.854009 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:44:09.854187 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:09.854191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:09.854252 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:09.854354 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:09.854846 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.855541 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.856960 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.857457 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.857549 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.857608 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:09.857955 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 13:44:09.858080 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 13:44:09.858315 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.853937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:35269 (local address 127.1.49.1:34257)
0926 13:44:09.854104 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:09.854111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:09.854491 (+ 380us) server_negotiation.cc:407] Connection header received
0926 13:44:09.854634 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:09.854638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:09.854689 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:09.854770 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:09.855140 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.855672 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.857059 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.857589 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.857727 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.857804 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:09.858079 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 13:44:09.858176 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:44:09.858731 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.853768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:34257 (local address 127.1.49.2:35269)
0926 13:44:09.854363 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:09.854379 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:09.854519 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:09.854842 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:09.854845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:09.854865 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:09.855010 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.855021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.855789 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.855792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.856856 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:09.856866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.857798 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.857800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:09.857932 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.857957 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 13:44:09.858041 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 13:44:09.858112 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":53,"spinlock_wait_cycles":33920,"thread_start_us":301,"threads_started":1}
I20260926 13:44:09.859045 19735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45eff10609aa4fa1ab2c13bed419f485" candidate_uuid: "df8132b7c0b244479c4b7a6d431c1219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43afb6c03bce4a509c28cc52a0db2d3c" is_pre_election: true
I20260926 13:44:09.859366 19467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45eff10609aa4fa1ab2c13bed419f485" candidate_uuid: "df8132b7c0b244479c4b7a6d431c1219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09eddbf874f4fd4a852be7d42f4216e" is_pre_election: true
I20260926 13:44:09.859402 19735 raft_consensus.cc:2454] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8132b7c0b244479c4b7a6d431c1219 in term 0.
I20260926 13:44:09.859711 19467 raft_consensus.cc:2454] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8132b7c0b244479c4b7a6d431c1219 in term 0.
I20260926 13:44:09.860051 19534 leader_election.cc:302] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43afb6c03bce4a509c28cc52a0db2d3c, df8132b7c0b244479c4b7a6d431c1219; no voters:
I20260926 13:44:09.860430 19806 raft_consensus.cc:2781] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:44:09.860536 19806 raft_consensus.cc:482] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:44:09.860600 19806 raft_consensus.cc:3037] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:09.861811 19806 raft_consensus.cc:504] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afb6c03bce4a509c28cc52a0db2d3c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 46299 } } peers { permanent_uuid: "b09eddbf874f4fd4a852be7d42f4216e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34257 } } peers { permanent_uuid: "df8132b7c0b244479c4b7a6d431c1219" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35721 } }
I20260926 13:44:09.862239 19806 leader_election.cc:288] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [CANDIDATE]: Term 1 election: Requested vote from peers 43afb6c03bce4a509c28cc52a0db2d3c (127.1.49.3:46299), b09eddbf874f4fd4a852be7d42f4216e (127.1.49.1:34257)
I20260926 13:44:09.862787 19735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45eff10609aa4fa1ab2c13bed419f485" candidate_uuid: "df8132b7c0b244479c4b7a6d431c1219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43afb6c03bce4a509c28cc52a0db2d3c"
I20260926 13:44:09.862787 19467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45eff10609aa4fa1ab2c13bed419f485" candidate_uuid: "df8132b7c0b244479c4b7a6d431c1219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09eddbf874f4fd4a852be7d42f4216e"
I20260926 13:44:09.862949 19735 raft_consensus.cc:3037] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:09.862949 19467 raft_consensus.cc:3037] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:09.864579 19735 raft_consensus.cc:2454] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8132b7c0b244479c4b7a6d431c1219 in term 1.
I20260926 13:44:09.864696 19467 raft_consensus.cc:2454] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8132b7c0b244479c4b7a6d431c1219 in term 1.
I20260926 13:44:09.865042 19537 leader_election.cc:302] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09eddbf874f4fd4a852be7d42f4216e, df8132b7c0b244479c4b7a6d431c1219; no voters:
I20260926 13:44:09.865304 19806 raft_consensus.cc:2781] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:09.865548 19806 raft_consensus.cc:686] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [term 1 LEADER]: Becoming Leader. State: Replica: df8132b7c0b244479c4b7a6d431c1219, State: Running, Role: LEADER
I20260926 13:44:09.865957 19806 consensus_queue.cc:237] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afb6c03bce4a509c28cc52a0db2d3c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 46299 } } peers { permanent_uuid: "b09eddbf874f4fd4a852be7d42f4216e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34257 } } peers { permanent_uuid: "df8132b7c0b244479c4b7a6d431c1219" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35721 } }
I20260926 13:44:09.869804 19322 catalog_manager.cc:5495] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8132b7c0b244479c4b7a6d431c1219 (127.1.49.2). New cstate: current_term: 1 leader_uuid: "df8132b7c0b244479c4b7a6d431c1219" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43afb6c03bce4a509c28cc52a0db2d3c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b09eddbf874f4fd4a852be7d42f4216e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8132b7c0b244479c4b7a6d431c1219" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 35721 } health_report { overall_health: HEALTHY } } }
W20260926 13:44:09.903650 19782 tablet.cc:2369] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:44:09.905664 19514 tablet.cc:2369] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:44:09.908121 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.904010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.1.49.2:35721)
0926 13:44:09.904144 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:09.904149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:09.904199 (+ 50us) server_negotiation.cc:407] Connection header received
0926 13:44:09.904385 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:09.904389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:09.904471 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:09.904572 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:09.904866 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.905451 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.906488 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:09.906793 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:09.906896 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:09.907194 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:09.907624 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:09.907907 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:09.907959 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 13:44:09.907998 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 13:44:09.908550 19648 tablet.cc:2369] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:44:09.914860 19467 raft_consensus.cc:1261] T 45eff10609aa4fa1ab2c13bed419f485 P b09eddbf874f4fd4a852be7d42f4216e [term 1 FOLLOWER]: Refusing update from remote peer df8132b7c0b244479c4b7a6d431c1219: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:09.914892 19735 raft_consensus.cc:1261] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c [term 1 FOLLOWER]: Refusing update from remote peer df8132b7c0b244479c4b7a6d431c1219: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:09.915777 19808 consensus_queue.cc:1035] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43afb6c03bce4a509c28cc52a0db2d3c" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:09.915998 19806 consensus_queue.cc:1035] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09eddbf874f4fd4a852be7d42f4216e" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:09.926496 19812 mvcc.cc:205] Tried to move back new op lower bound from 7333602303642910720 to 7333602303454355456. Current Snapshot: MvccSnapshot[applied={T|T < 7333602303642910720}]
I20260926 13:44:09.936004 19321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:09.936235 19321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:09.939057 19321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f78e817fc68143728404eb75e27582f8: Sending DeleteTablet for 3 replicas of tablet 45eff10609aa4fa1ab2c13bed419f485
I20260926 13:44:09.940028 19715 tablet_service.cc:1514] Processing DeleteTablet for tablet 45eff10609aa4fa1ab2c13bed419f485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:09 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:44524
I20260926 13:44:09.940138 19447 tablet_service.cc:1514] Processing DeleteTablet for tablet 45eff10609aa4fa1ab2c13bed419f485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:09 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:58126
I20260926 13:44:09.940667 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 19384
I20260926 13:44:09.942662 19581 tablet_service.cc:1514] Processing DeleteTablet for tablet 45eff10609aa4fa1ab2c13bed419f485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:09 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:51164
I20260926 13:44:09.943245 19823 tablet_replica.cc:330] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219: stopping tablet replica
I20260926 13:44:09.943652 19823 raft_consensus.cc:2229] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:44:09.944043 19821 tablet_replica.cc:330] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c: stopping tablet replica
I20260926 13:44:09.944178 19823 raft_consensus.cc:2258] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:09.944805 19821 raft_consensus.cc:2229] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:44:09.945214 19821 raft_consensus.cc:2258] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:09.945685 19823 ts_tablet_manager.cc:1905] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:09.946928 19821 ts_tablet_manager.cc:1905] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 13:44:09.949096 19309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:34257 (error 108)
I20260926 13:44:09.949501 19823 ts_tablet_manager.cc:1918] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:44:09.949617 19823 log.cc:1196] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/45eff10609aa4fa1ab2c13bed419f485
I20260926 13:44:09.949961 19823 ts_tablet_manager.cc:1939] T 45eff10609aa4fa1ab2c13bed419f485 P df8132b7c0b244479c4b7a6d431c1219: Deleting consensus metadata
I20260926 13:44:09.950062 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 19518
I20260926 13:44:09.950423 19821 ts_tablet_manager.cc:1918] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:44:09.950673 19821 log.cc:1196] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/45eff10609aa4fa1ab2c13bed419f485
I20260926 13:44:09.951311 19821 ts_tablet_manager.cc:1939] T 45eff10609aa4fa1ab2c13bed419f485 P 43afb6c03bce4a509c28cc52a0db2d3c: Deleting consensus metadata
I20260926 13:44:09.952840 19306 catalog_manager.cc:4841] TS 43afb6c03bce4a509c28cc52a0db2d3c (127.1.49.3:46299): tablet 45eff10609aa4fa1ab2c13bed419f485 (table test-table [id=425a173230b94cb1a787e44f07140de9]) successfully deleted
I20260926 13:44:09.953285 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.952893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:34257 (local address 127.0.0.1:58138)
0926 13:44:09.953071 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:09.953162 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:34257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 13:44:09.953476 19309 catalog_manager.cc:4584] TS b09eddbf874f4fd4a852be7d42f4216e (127.1.49.1:34257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45eff10609aa4fa1ab2c13bed419f485: Network error: Client connection negotiation failed: client connection to 127.1.49.1:34257: connect: Connection refused (error 111)
I20260926 13:44:09.956353 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 19652
I20260926 13:44:09.958015 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:09.956591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:35721 (local address 127.0.0.1:51192)
0926 13:44:09.957833 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:09.957906 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.2:35721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1133}
I20260926 13:44:09.962200 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 19290
2026-09-26T13:44:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2576 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T13:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:09Z Disabled control of system clock
I20260926 13:44:09.995121 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:38825
--webserver_interface=127.1.49.62
--webserver_port=0
--builtin_ntp_servers=127.1.49.20:37937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:38825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:44:10.118093 19828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:10.118443 19828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:10.118485 19828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:10.123407 19828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:10.123489 19828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:10.123505 19828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:10.123525 19828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:10.123541 19828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:10.128779 19828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:38825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:38825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:10.130029 19828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:10.131654 19828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:10.141418 19836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:10.141467 19833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:10.141441 19834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:10.141903 19828 server_base.cc:1034] running on GCE node
I20260926 13:44:10.142678 19828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:10.144042 19828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:10.145257 19828 hybrid_clock.cc:648] HybridClock initialized: now 1790430250145239 us; error 34 us; skew 500 ppm
I20260926 13:44:10.147434 19828 webserver.cc:460] Webserver started at http://127.1.49.62:36575/ using document root <none> and password file <none>
I20260926 13:44:10.148157 19828 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:10.148274 19828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:10.148552 19828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:10.150858 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "891d7812c53441088ddbd2b0a381f9a2"
format_stamp: "Formatted at 2026-09-26 13:44:10 on dist-test-slave-1wns"
server_key: "50343b2343bd4d8efba15b584b2d682d"
server_key_iv: "2198bcd8516a88f7be826897882f935a"
server_key_version: "encryptionkey@0"
I20260926 13:44:10.151508 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "891d7812c53441088ddbd2b0a381f9a2"
format_stamp: "Formatted at 2026-09-26 13:44:10 on dist-test-slave-1wns"
server_key: "50343b2343bd4d8efba15b584b2d682d"
server_key_iv: "2198bcd8516a88f7be826897882f935a"
server_key_version: "encryptionkey@0"
I20260926 13:44:10.155781 19828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:44:10.158650 19842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:10.160174 19828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 13:44:10.160358 19828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "891d7812c53441088ddbd2b0a381f9a2"
format_stamp: "Formatted at 2026-09-26 13:44:10 on dist-test-slave-1wns"
server_key: "50343b2343bd4d8efba15b584b2d682d"
server_key_iv: "2198bcd8516a88f7be826897882f935a"
server_key_version: "encryptionkey@0"
I20260926 13:44:10.160494 19828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:10.174460 19828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:10.175261 19828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:10.175486 19828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:10.182898 19828 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:38825
I20260926 13:44:10.182911 19894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:38825 every 8 connection(s)
I20260926 13:44:10.183938 19828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:10.187458 19895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:10.192723 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 19828
I20260926 13:44:10.192847 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:10.193151 1220 external_mini_cluster.cc:1428] Setting key 7a1e1109699767a4d18b717261074207
I20260926 13:44:10.194296 19895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2: Bootstrap starting.
I20260926 13:44:10.196897 19895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:10.197939 19895 log.cc:824] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:10.200141 19895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2: No bootstrap required, opened a new log
I20260926 13:44:10.203991 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:10.194809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.1.49.62:38825)
0926 13:44:10.195497 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:10.195513 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:10.195550 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:44:10.196358 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:10.196380 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:10.196828 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:10.197208 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:10.197599 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.198659 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.199956 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.200525 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.200728 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:10.201773 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:10.201783 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:10.201805 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:10.202270 (+ 465us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:10.202336 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:10.202358 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:10.202488 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:10.202678 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 13:44:10.202998 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":160,"threads_started":1}
I20260926 13:44:10.204566 19895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891d7812c53441088ddbd2b0a381f9a2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 38825 } }
I20260926 13:44:10.204895 19895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:10.204979 19895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891d7812c53441088ddbd2b0a381f9a2, State: Initialized, Role: FOLLOWER
I20260926 13:44:10.205485 19895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891d7812c53441088ddbd2b0a381f9a2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 38825 } }
I20260926 13:44:10.205621 19895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:10.205720 19895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:10.205853 19895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:10.207122 19895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891d7812c53441088ddbd2b0a381f9a2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 38825 } }
I20260926 13:44:10.207513 19895 leader_election.cc:302] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 891d7812c53441088ddbd2b0a381f9a2; no voters:
I20260926 13:44:10.207931 19895 leader_election.cc:288] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:10.208067 19900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:10.208354 19900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [term 1 LEADER]: Becoming Leader. State: Replica: 891d7812c53441088ddbd2b0a381f9a2, State: Running, Role: LEADER
I20260926 13:44:10.208705 19900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891d7812c53441088ddbd2b0a381f9a2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 38825 } }
I20260926 13:44:10.209379 19895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:10.210547 19902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 891d7812c53441088ddbd2b0a381f9a2. Latest consensus state: current_term: 1 leader_uuid: "891d7812c53441088ddbd2b0a381f9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891d7812c53441088ddbd2b0a381f9a2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 38825 } } }
I20260926 13:44:10.210695 19902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:10.211251 19901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "891d7812c53441088ddbd2b0a381f9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891d7812c53441088ddbd2b0a381f9a2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 38825 } } }
I20260926 13:44:10.211911 19901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2 [sys.catalog]: This master's current role is: LEADER
W20260926 13:44:10.214843 19916 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:10.214959 19916 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:10.215037 19909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:10.216111 19909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:10.222472 19909 catalog_manager.cc:1347] Generated new cluster ID: adcf4f79a7b1403b9b210e73972b3427
I20260926 13:44:10.222572 19909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:10.261641 19909 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:10.262993 19909 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:10.275674 19909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2: Generated new TSK 0
I20260926 13:44:10.276682 19909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:10.288197 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:38825
--builtin_ntp_servers=127.1.49.20:37937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:44:10.405975 19919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:10.406324 19919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:10.406406 19919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:10.410897 19919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:10.410997 19919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:10.411079 19919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:44:10.416194 19919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:38825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:10.417347 19919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:10.418766 19919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:10.429364 19925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:10.429433 19927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:10.429463 19924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:10.429839 19919 server_base.cc:1034] running on GCE node
I20260926 13:44:10.430307 19919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:10.431015 19919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:10.432246 19919 hybrid_clock.cc:648] HybridClock initialized: now 1790430250432223 us; error 35 us; skew 500 ppm
I20260926 13:44:10.434259 19919 webserver.cc:460] Webserver started at http://127.1.49.1:43243/ using document root <none> and password file <none>
I20260926 13:44:10.434948 19919 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:10.435041 19919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:10.435289 19919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:10.437533 19919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "70bba6b2b7e34a338c1b1d573e5d7012"
format_stamp: "Formatted at 2026-09-26 13:44:10 on dist-test-slave-1wns"
server_key: "4b1b1cbf0ef96bc39c8b1801bfe134c7"
server_key_iv: "4adcd1110eb4a106e86cdac1fa12a327"
server_key_version: "encryptionkey@0"
I20260926 13:44:10.438131 19919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "70bba6b2b7e34a338c1b1d573e5d7012"
format_stamp: "Formatted at 2026-09-26 13:44:10 on dist-test-slave-1wns"
server_key: "4b1b1cbf0ef96bc39c8b1801bfe134c7"
server_key_iv: "4adcd1110eb4a106e86cdac1fa12a327"
server_key_version: "encryptionkey@0"
I20260926 13:44:10.442140 19919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 13:44:10.444846 19933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:10.446275 19919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:10.446460 19919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "70bba6b2b7e34a338c1b1d573e5d7012"
format_stamp: "Formatted at 2026-09-26 13:44:10 on dist-test-slave-1wns"
server_key: "4b1b1cbf0ef96bc39c8b1801bfe134c7"
server_key_iv: "4adcd1110eb4a106e86cdac1fa12a327"
server_key_version: "encryptionkey@0"
I20260926 13:44:10.446599 19919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:10.466120 19919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:10.466943 19919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:10.467173 19919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:10.467947 19919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:10.469033 19919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:10.469112 19919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:10.469194 19919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:10.469251 19919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:10.479559 19919 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:45549
I20260926 13:44:10.479597 20046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:45549 every 8 connection(s)
I20260926 13:44:10.480705 19919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:44:10.485688 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 19919
I20260926 13:44:10.485882 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:44:10.486284 1220 external_mini_cluster.cc:1428] Setting key 6131369524d341e9b6a1322b95cb1eed
I20260926 13:44:10.488560 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:38825
--builtin_ntp_servers=127.1.49.20:37937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:44:10.491916 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:10.482944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:35349 (local address 127.1.49.62:38825)
0926 13:44:10.483116 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:10.483122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:10.484516 (+ 1394us) server_negotiation.cc:407] Connection header received
0926 13:44:10.486192 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:10.486196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:10.486274 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:10.486388 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:10.487522 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.488175 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.489624 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.489888 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.490015 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:10.491306 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:10.491310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:10.491313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:10.491329 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:10.491360 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:10.491369 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:10.491454 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:10.491657 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 13:44:10.491730 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 13:44:10.492870 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:10.483529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:38825 (local address 127.1.49.1:35349)
0926 13:44:10.484338 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:10.484397 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:10.485919 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:10.486593 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:10.486600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:10.486698 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:10.487367 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:10.487384 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.488320 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.488324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:10.489482 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:10.489492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.490061 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.490065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:10.490259 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:10.490987 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:44:10.490988 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:10.491034 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:44:10.491118 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:10.491470 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:10.491483 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:10.491491 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 13:44:10.491642 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 13:44:10.491996 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":186,"threads_started":1}
I20260926 13:44:10.494338 20047 heartbeater.cc:347] Connected to a master server at 127.1.49.62:38825
I20260926 13:44:10.494660 20047 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:10.495560 20047 heartbeater.cc:511] Master 127.1.49.62:38825 requested a full tablet report, sending...
I20260926 13:44:10.497730 19859 ts_manager.cc:194] Registered new tserver with Master: 70bba6b2b7e34a338c1b1d573e5d7012 (127.1.49.1:45549)
I20260926 13:44:10.499766 19859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.49.1:35349
W20260926 13:44:10.611661 20050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:10.612023 20050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:10.612069 20050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:10.616380 20050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:10.616453 20050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:10.616513 20050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:44:10.621593 20050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:38825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:10.622740 20050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:10.624221 20050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:10.634886 20058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:10.634932 20050 server_base.cc:1034] running on GCE node
W20260926 13:44:10.634891 20055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:10.634881 20056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:10.635643 20050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:10.636443 20050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:10.637692 20050 hybrid_clock.cc:648] HybridClock initialized: now 1790430250637647 us; error 55 us; skew 500 ppm
I20260926 13:44:10.640162 20050 webserver.cc:460] Webserver started at http://127.1.49.2:39473/ using document root <none> and password file <none>
I20260926 13:44:10.640856 20050 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:10.640969 20050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:10.641258 20050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:10.643682 20050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "f25a39a5853c463480357102102b85a0"
format_stamp: "Formatted at 2026-09-26 13:44:10 on dist-test-slave-1wns"
server_key: "196aa00da97282aabe7e89846a109faa"
server_key_iv: "b34477c5c6a41083123273476d39cf61"
server_key_version: "encryptionkey@0"
I20260926 13:44:10.644464 20050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "f25a39a5853c463480357102102b85a0"
format_stamp: "Formatted at 2026-09-26 13:44:10 on dist-test-slave-1wns"
server_key: "196aa00da97282aabe7e89846a109faa"
server_key_iv: "b34477c5c6a41083123273476d39cf61"
server_key_version: "encryptionkey@0"
I20260926 13:44:10.649183 20050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:44:10.652431 20064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:10.653936 20050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:44:10.654119 20050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "f25a39a5853c463480357102102b85a0"
format_stamp: "Formatted at 2026-09-26 13:44:10 on dist-test-slave-1wns"
server_key: "196aa00da97282aabe7e89846a109faa"
server_key_iv: "b34477c5c6a41083123273476d39cf61"
server_key_version: "encryptionkey@0"
I20260926 13:44:10.654290 20050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:10.678177 20050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:10.679044 20050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:10.679237 20050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:10.679944 20050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:10.681052 20050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:10.681111 20050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:10.681162 20050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:10.681182 20050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:10.693872 20177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:37175 every 8 connection(s)
I20260926 13:44:10.693873 20050 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:37175
I20260926 13:44:10.695171 20050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:44:10.696547 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 20050
I20260926 13:44:10.696705 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:44:10.697109 1220 external_mini_cluster.cc:1428] Setting key 33408a278358a8809454a3ae403ab580
I20260926 13:44:10.699729 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:38825
--builtin_ntp_servers=127.1.49.20:37937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:44:10.707624 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:10.697485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:50811 (local address 127.1.49.62:38825)
0926 13:44:10.697659 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:10.697665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:10.699430 (+ 1765us) server_negotiation.cc:407] Connection header received
0926 13:44:10.701623 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:10.701629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:10.701700 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:10.701819 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:10.702855 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.703771 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.705138 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.705576 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.705714 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:10.706983 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:10.706990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:10.706996 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:10.707018 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:10.707048 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:10.707061 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:10.707153 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:10.707314 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 13:44:10.707399 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 13:44:10.708612 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:10.698134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:38825 (local address 127.1.49.2:50811)
0926 13:44:10.699246 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:10.699310 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:10.701325 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:10.702001 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:10.702008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:10.702103 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:10.702655 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:10.702668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.703935 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.703938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:10.704979 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:10.704990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.705814 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.705817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:10.705986 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:10.706675 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:44:10.706677 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:10.706729 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:44:10.706805 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:10.707159 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:10.707166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:10.707170 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:44:10.707278 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 13:44:10.707543 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":184,"threads_started":1}
I20260926 13:44:10.709978 20178 heartbeater.cc:347] Connected to a master server at 127.1.49.62:38825
I20260926 13:44:10.710276 20178 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:10.711062 20178 heartbeater.cc:511] Master 127.1.49.62:38825 requested a full tablet report, sending...
I20260926 13:44:10.712523 19859 ts_manager.cc:194] Registered new tserver with Master: f25a39a5853c463480357102102b85a0 (127.1.49.2:37175)
I20260926 13:44:10.713311 19859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.49.2:50811
W20260926 13:44:10.821995 20181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:10.822278 20181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:10.822319 20181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:10.826439 20181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:10.826501 20181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:10.826558 20181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:44:10.831454 20181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:38825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:10.832592 20181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:10.834003 20181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:10.844014 20189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:10.844000 20187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:10.844022 20186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:10.844400 20181 server_base.cc:1034] running on GCE node
I20260926 13:44:10.844820 20181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:10.845501 20181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:10.846688 20181 hybrid_clock.cc:648] HybridClock initialized: now 1790430250846652 us; error 49 us; skew 500 ppm
I20260926 13:44:10.849153 20181 webserver.cc:460] Webserver started at http://127.1.49.3:44933/ using document root <none> and password file <none>
I20260926 13:44:10.849838 20181 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:10.849931 20181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:10.850142 20181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:10.852350 20181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "025dfdeef84047cbbff6e4d28a4e0b27"
format_stamp: "Formatted at 2026-09-26 13:44:10 on dist-test-slave-1wns"
server_key: "b945243b5dd749e8973a19a86306a58b"
server_key_iv: "d903ce0f1d930dba9e3d007d598418e7"
server_key_version: "encryptionkey@0"
I20260926 13:44:10.853045 20181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "025dfdeef84047cbbff6e4d28a4e0b27"
format_stamp: "Formatted at 2026-09-26 13:44:10 on dist-test-slave-1wns"
server_key: "b945243b5dd749e8973a19a86306a58b"
server_key_iv: "d903ce0f1d930dba9e3d007d598418e7"
server_key_version: "encryptionkey@0"
I20260926 13:44:10.857013 20181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 13:44:10.860056 20195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:10.861581 20181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 13:44:10.861729 20181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "025dfdeef84047cbbff6e4d28a4e0b27"
format_stamp: "Formatted at 2026-09-26 13:44:10 on dist-test-slave-1wns"
server_key: "b945243b5dd749e8973a19a86306a58b"
server_key_iv: "d903ce0f1d930dba9e3d007d598418e7"
server_key_version: "encryptionkey@0"
I20260926 13:44:10.861862 20181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:10.881940 20181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:10.882716 20181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:10.882941 20181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:10.883571 20181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:10.884665 20181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:10.884750 20181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:10.884826 20181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:10.884871 20181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:10.894579 20181 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:43965
I20260926 13:44:10.894650 20308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:43965 every 8 connection(s)
I20260926 13:44:10.895633 20181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:44:10.898360 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 20181
I20260926 13:44:10.898486 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:44:10.898877 1220 external_mini_cluster.cc:1428] Setting key 936f0e1177fd63c2bd103382492c8fa1
I20260926 13:44:10.906395 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:10.897805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:40199 (local address 127.1.49.62:38825)
0926 13:44:10.897967 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:10.897971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:10.899469 (+ 1498us) server_negotiation.cc:407] Connection header received
0926 13:44:10.901206 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:10.901210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:10.901290 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:10.901378 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:10.902283 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.902889 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.904299 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.904539 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.904631 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:10.905944 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:10.905947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:10.905949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:10.905963 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:10.905983 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:10.905991 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:10.906052 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:10.906203 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 13:44:10.906275 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 13:44:10.907380 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:10.898367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:38825 (local address 127.1.49.3:40199)
0926 13:44:10.899243 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:10.899319 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:10.900959 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:10.901532 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:10.901539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:10.901628 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:10.902137 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:10.902150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.903033 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.903037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:10.904162 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:10.904174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.904761 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.904765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:10.904926 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:10.905665 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:44:10.905667 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:10.905710 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:44:10.905781 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:10.906070 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:10.906077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:10.906081 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:44:10.906191 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 13:44:10.906433 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":179,"threads_started":1}
I20260926 13:44:10.908720 20309 heartbeater.cc:347] Connected to a master server at 127.1.49.62:38825
I20260926 13:44:10.909106 20309 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:10.909677 20309 heartbeater.cc:511] Master 127.1.49.62:38825 requested a full tablet report, sending...
I20260926 13:44:10.910826 19859 ts_manager.cc:194] Registered new tserver with Master: 025dfdeef84047cbbff6e4d28a4e0b27 (127.1.49.3:43965)
I20260926 13:44:10.911494 19859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.49.3:40199
I20260926 13:44:10.913308 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:44:10.919745 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:10.915504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.1.49.62:38825)
0926 13:44:10.915679 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:10.915685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:10.915863 (+ 178us) server_negotiation.cc:407] Connection header received
0926 13:44:10.916054 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:10.916057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:10.916109 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:10.916196 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:10.916491 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.917042 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.917910 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.918151 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.918264 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:10.919214 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:10.919217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:10.919220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:10.919237 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:10.919261 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:10.919270 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:10.919404 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:10.919537 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 13:44:10.919580 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 13:44:10.923146 19859 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:44:10.926513 19859 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:44:10.943296 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:10.937108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:45549 (local address 127.0.0.1:57664)
0926 13:44:10.938075 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:10.938093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:10.938201 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:10.938852 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:10.938855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:10.938878 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:10.939017 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:10.939025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.940621 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.940625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:10.941792 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:10.941802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.942671 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.942675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:10.942795 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:10.942837 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 13:44:10.942967 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 13:44:10.943108 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":53,"thread_start_us":102,"threads_started":1}
I20260926 13:44:10.943302 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:10.937611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.1.49.1:45549)
0926 13:44:10.938288 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:10.938303 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:10.938327 (+ 24us) server_negotiation.cc:407] Connection header received
0926 13:44:10.938411 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:10.938418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:10.938653 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:10.938841 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:10.939174 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.940479 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.941929 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.942506 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.942674 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:10.942903 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:10.943000 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 13:44:10.943120 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":105,"threads_started":1}
I20260926 13:44:10.943586 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:10.936673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:43965 (local address 127.0.0.1:41256)
0926 13:44:10.937421 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:10.937477 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:10.937623 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:10.938763 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:10.938772 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:10.938802 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:10.938976 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:10.938986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.940237 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.940244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:10.941673 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:10.941685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.943114 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.943118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:10.943216 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:10.943240 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 13:44:10.943316 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 13:44:10.943393 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":93,"threads_started":1}
I20260926 13:44:10.944564 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:10.936821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.1.49.3:43965)
0926 13:44:10.938204 (+ 1383us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:10.938219 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:10.938247 (+ 28us) server_negotiation.cc:407] Connection header received
0926 13:44:10.938343 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:10.938349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:10.938587 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:10.938734 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:10.939127 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.940115 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.941829 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.942741 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.942857 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:10.943045 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:10.943889 (+ 844us) server_negotiation.cc:299] Negotiation successful
0926 13:44:10.944011 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1263,"thread_start_us":169,"threads_started":1}
I20260926 13:44:10.944622 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:10.936781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:37175 (local address 127.0.0.1:46232)
0926 13:44:10.937794 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:10.937809 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:10.937908 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:10.939631 (+ 1723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:10.939644 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:10.939666 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:10.939788 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:10.939795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.941274 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.941277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:10.942611 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:10.942623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.944115 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.944119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:10.944239 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:10.944267 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 13:44:10.944393 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 13:44:10.944471 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":4,"thread_start_us":125,"threads_started":1}
I20260926 13:44:10.945168 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:10.937611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.1.49.2:37175)
0926 13:44:10.939133 (+ 1522us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:10.939149 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:10.939173 (+ 24us) server_negotiation.cc:407] Connection header received
0926 13:44:10.939253 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:10.939258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:10.939467 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:10.939629 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:10.939961 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.941137 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.942779 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:10.943979 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:10.944098 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:10.944292 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:10.944886 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 13:44:10.944996 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1385,"thread_start_us":116,"threads_started":1}
I20260926 13:44:10.946692 19981 tablet_service.cc:1467] Processing CreateTablet for tablet 17318f1dab3b447ab09eba5eab344867 (DEFAULT_TABLE table=test-table [id=d2cc6a112b5f48c68da69d3330d37832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:10.947969 19981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17318f1dab3b447ab09eba5eab344867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:10.947551 20243 tablet_service.cc:1467] Processing CreateTablet for tablet 17318f1dab3b447ab09eba5eab344867 (DEFAULT_TABLE table=test-table [id=d2cc6a112b5f48c68da69d3330d37832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:10.948673 20243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17318f1dab3b447ab09eba5eab344867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:10.949020 20112 tablet_service.cc:1467] Processing CreateTablet for tablet 17318f1dab3b447ab09eba5eab344867 (DEFAULT_TABLE table=test-table [id=d2cc6a112b5f48c68da69d3330d37832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:10.950076 20112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17318f1dab3b447ab09eba5eab344867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:10.956992 20326 tablet_bootstrap.cc:492] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27: Bootstrap starting.
I20260926 13:44:10.957497 20327 tablet_bootstrap.cc:492] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012: Bootstrap starting.
I20260926 13:44:10.958204 20328 tablet_bootstrap.cc:492] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0: Bootstrap starting.
I20260926 13:44:10.960035 20327 tablet_bootstrap.cc:654] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:10.960109 20326 tablet_bootstrap.cc:654] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:10.960608 20328 tablet_bootstrap.cc:654] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:10.960903 20327 log.cc:824] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:10.961020 20326 log.cc:824] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:10.961681 20328 log.cc:824] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:10.963398 20326 tablet_bootstrap.cc:492] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27: No bootstrap required, opened a new log
I20260926 13:44:10.963413 20327 tablet_bootstrap.cc:492] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012: No bootstrap required, opened a new log
I20260926 13:44:10.963634 20327 ts_tablet_manager.cc:1397] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260926 13:44:10.963634 20326 ts_tablet_manager.cc:1397] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 13:44:10.963857 20328 tablet_bootstrap.cc:492] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0: No bootstrap required, opened a new log
I20260926 13:44:10.964074 20328 ts_tablet_manager.cc:1397] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:44:10.968276 20327 raft_consensus.cc:348] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025dfdeef84047cbbff6e4d28a4e0b27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 43965 } } peers { permanent_uuid: "f25a39a5853c463480357102102b85a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37175 } } peers { permanent_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45549 } }
I20260926 13:44:10.968606 20327 raft_consensus.cc:374] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:10.968673 20327 raft_consensus.cc:729] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70bba6b2b7e34a338c1b1d573e5d7012, State: Initialized, Role: FOLLOWER
I20260926 13:44:10.968865 20326 raft_consensus.cc:348] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025dfdeef84047cbbff6e4d28a4e0b27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 43965 } } peers { permanent_uuid: "f25a39a5853c463480357102102b85a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37175 } } peers { permanent_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45549 } }
I20260926 13:44:10.968870 20328 raft_consensus.cc:348] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025dfdeef84047cbbff6e4d28a4e0b27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 43965 } } peers { permanent_uuid: "f25a39a5853c463480357102102b85a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37175 } } peers { permanent_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45549 } }
I20260926 13:44:10.969174 20326 raft_consensus.cc:374] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:10.969174 20328 raft_consensus.cc:374] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:10.969239 20326 raft_consensus.cc:729] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025dfdeef84047cbbff6e4d28a4e0b27, State: Initialized, Role: FOLLOWER
I20260926 13:44:10.969238 20328 raft_consensus.cc:729] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f25a39a5853c463480357102102b85a0, State: Initialized, Role: FOLLOWER
I20260926 13:44:10.969237 20327 consensus_queue.cc:260] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025dfdeef84047cbbff6e4d28a4e0b27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 43965 } } peers { permanent_uuid: "f25a39a5853c463480357102102b85a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37175 } } peers { permanent_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45549 } }
I20260926 13:44:10.969765 20326 consensus_queue.cc:260] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025dfdeef84047cbbff6e4d28a4e0b27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 43965 } } peers { permanent_uuid: "f25a39a5853c463480357102102b85a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37175 } } peers { permanent_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45549 } }
I20260926 13:44:10.969764 20328 consensus_queue.cc:260] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025dfdeef84047cbbff6e4d28a4e0b27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 43965 } } peers { permanent_uuid: "f25a39a5853c463480357102102b85a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37175 } } peers { permanent_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45549 } }
I20260926 13:44:10.970168 20047 heartbeater.cc:503] Master 127.1.49.62:38825 was elected leader, sending a full tablet report...
I20260926 13:44:10.970634 20309 heartbeater.cc:503] Master 127.1.49.62:38825 was elected leader, sending a full tablet report...
I20260926 13:44:10.970794 20178 heartbeater.cc:503] Master 127.1.49.62:38825 was elected leader, sending a full tablet report...
I20260926 13:44:10.970920 20328 ts_tablet_manager.cc:1428] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 13:44:10.971732 20327 ts_tablet_manager.cc:1428] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012: Time spent starting tablet: real 0.008s user 0.002s sys 0.006s
I20260926 13:44:10.973358 20326 ts_tablet_manager.cc:1428] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
W20260926 13:44:10.982040 20048 tablet.cc:2369] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:44:11.017369 20332 raft_consensus.cc:482] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:44:11.017609 20332 raft_consensus.cc:504] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025dfdeef84047cbbff6e4d28a4e0b27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 43965 } } peers { permanent_uuid: "f25a39a5853c463480357102102b85a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37175 } } peers { permanent_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45549 } }
I20260926 13:44:11.018908 20332 leader_election.cc:288] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 025dfdeef84047cbbff6e4d28a4e0b27 (127.1.49.3:43965), f25a39a5853c463480357102102b85a0 (127.1.49.2:37175)
I20260926 13:44:11.023693 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:11.019124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:43965 (local address 127.1.49.1:38117)
0926 13:44:11.019467 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:11.019486 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:11.019632 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:11.019965 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:11.019969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:11.019994 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:11.020148 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:11.020160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:11.021212 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:11.021216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:11.022220 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:11.022230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:11.023171 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:11.023175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:11.023305 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:11.023333 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 13:44:11.023417 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 13:44:11.023510 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":141,"threads_started":1}
I20260926 13:44:11.023700 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:11.019190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:38117 (local address 127.1.49.3:43965)
0926 13:44:11.019388 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:11.019394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:11.019649 (+ 255us) server_negotiation.cc:407] Connection header received
0926 13:44:11.019740 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:11.019744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:11.019808 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:11.019949 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:11.020432 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:11.021079 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:11.022418 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:11.023036 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:11.023126 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:11.023181 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:11.023462 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 13:44:11.023548 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 13:44:11.024402 20263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17318f1dab3b447ab09eba5eab344867" candidate_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025dfdeef84047cbbff6e4d28a4e0b27" is_pre_election: true
I20260926 13:44:11.024634 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:11.019172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:37175 (local address 127.1.49.1:43923)
0926 13:44:11.019589 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:11.019603 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:11.019681 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:11.020380 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:11.020385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:11.020409 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:11.020553 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:11.020561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:11.021566 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:11.021569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:11.022749 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:11.022760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:11.023642 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:11.023646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:11.023749 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:11.024219 (+ 470us) client_negotiation.cc:769] Sending connection context
0926 13:44:11.024333 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 13:44:11.024403 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":147,"threads_started":1}
I20260926 13:44:11.024631 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:11.019768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:43923 (local address 127.1.49.2:37175)
0926 13:44:11.019977 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:11.019982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:11.019996 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:44:11.020050 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:11.020055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:11.020112 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:11.020204 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:11.020787 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:11.021432 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:11.022889 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:11.023426 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:11.023518 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:11.023571 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:11.024384 (+ 813us) server_negotiation.cc:299] Negotiation successful
0926 13:44:11.024493 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260926 13:44:11.024782 20263 raft_consensus.cc:2454] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70bba6b2b7e34a338c1b1d573e5d7012 in term 0.
I20260926 13:44:11.025307 19935 leader_election.cc:302] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025dfdeef84047cbbff6e4d28a4e0b27, 70bba6b2b7e34a338c1b1d573e5d7012; no voters:
I20260926 13:44:11.025331 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17318f1dab3b447ab09eba5eab344867" candidate_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f25a39a5853c463480357102102b85a0" is_pre_election: true
I20260926 13:44:11.025606 20332 raft_consensus.cc:2781] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:44:11.025722 20332 raft_consensus.cc:482] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:44:11.025677 20132 raft_consensus.cc:2454] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70bba6b2b7e34a338c1b1d573e5d7012 in term 0.
I20260926 13:44:11.025791 20332 raft_consensus.cc:3037] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:11.027096 20332 raft_consensus.cc:504] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025dfdeef84047cbbff6e4d28a4e0b27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 43965 } } peers { permanent_uuid: "f25a39a5853c463480357102102b85a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37175 } } peers { permanent_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45549 } }
I20260926 13:44:11.027521 20332 leader_election.cc:288] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [CANDIDATE]: Term 1 election: Requested vote from peers 025dfdeef84047cbbff6e4d28a4e0b27 (127.1.49.3:43965), f25a39a5853c463480357102102b85a0 (127.1.49.2:37175)
I20260926 13:44:11.027961 20132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17318f1dab3b447ab09eba5eab344867" candidate_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f25a39a5853c463480357102102b85a0"
I20260926 13:44:11.027972 20263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17318f1dab3b447ab09eba5eab344867" candidate_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025dfdeef84047cbbff6e4d28a4e0b27"
I20260926 13:44:11.028110 20132 raft_consensus.cc:3037] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:11.028141 20263 raft_consensus.cc:3037] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:11.029309 20263 raft_consensus.cc:2454] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70bba6b2b7e34a338c1b1d573e5d7012 in term 1.
I20260926 13:44:11.029322 20132 raft_consensus.cc:2454] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70bba6b2b7e34a338c1b1d573e5d7012 in term 1.
I20260926 13:44:11.029701 19935 leader_election.cc:302] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025dfdeef84047cbbff6e4d28a4e0b27, 70bba6b2b7e34a338c1b1d573e5d7012; no voters:
I20260926 13:44:11.029948 20332 raft_consensus.cc:2781] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:11.030157 20332 raft_consensus.cc:686] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [term 1 LEADER]: Becoming Leader. State: Replica: 70bba6b2b7e34a338c1b1d573e5d7012, State: Running, Role: LEADER
I20260926 13:44:11.030531 20332 consensus_queue.cc:237] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025dfdeef84047cbbff6e4d28a4e0b27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 43965 } } peers { permanent_uuid: "f25a39a5853c463480357102102b85a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37175 } } peers { permanent_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45549 } }
I20260926 13:44:11.034354 19858 catalog_manager.cc:5495] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70bba6b2b7e34a338c1b1d573e5d7012 (127.1.49.1). New cstate: current_term: 1 leader_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025dfdeef84047cbbff6e4d28a4e0b27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 43965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f25a39a5853c463480357102102b85a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70bba6b2b7e34a338c1b1d573e5d7012" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 45549 } health_report { overall_health: HEALTHY } } }
I20260926 13:44:11.054468 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:11.050056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.1.49.1:45549)
0926 13:44:11.050183 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:11.050186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:11.050199 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:44:11.050272 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:11.050274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:11.050330 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:11.050428 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:11.050829 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:11.051480 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:11.052603 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:11.052877 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:11.052975 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:11.053423 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:11.053982 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:11.054111 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:11.054305 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 13:44:11.054348 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:44:11.061123 20263 raft_consensus.cc:1261] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27 [term 1 FOLLOWER]: Refusing update from remote peer 70bba6b2b7e34a338c1b1d573e5d7012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:11.061123 20132 raft_consensus.cc:1261] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0 [term 1 FOLLOWER]: Refusing update from remote peer 70bba6b2b7e34a338c1b1d573e5d7012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:11.062156 20337 consensus_queue.cc:1035] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f25a39a5853c463480357102102b85a0" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:11.062464 20332 consensus_queue.cc:1035] T 17318f1dab3b447ab09eba5eab344867 P 70bba6b2b7e34a338c1b1d573e5d7012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025dfdeef84047cbbff6e4d28a4e0b27" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:11.074774 20341 mvcc.cc:205] Tried to move back new op lower bound from 7333602308338450432 to 7333602308224147456. Current Snapshot: MvccSnapshot[applied={T|T < 7333602308338450432 or (T in {7333602308338450432})}]
I20260926 13:44:11.081525 19858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:11.081710 19858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:11.084288 19858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 891d7812c53441088ddbd2b0a381f9a2: Sending DeleteTablet for 3 replicas of tablet 17318f1dab3b447ab09eba5eab344867
I20260926 13:44:11.085188 20112 tablet_service.cc:1514] Processing DeleteTablet for tablet 17318f1dab3b447ab09eba5eab344867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:11 UTC) from {username='slave'} at 127.0.0.1:46232
I20260926 13:44:11.085187 19981 tablet_service.cc:1514] Processing DeleteTablet for tablet 17318f1dab3b447ab09eba5eab344867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:11 UTC) from {username='slave'} at 127.0.0.1:57664
I20260926 13:44:11.085170 20243 tablet_service.cc:1514] Processing DeleteTablet for tablet 17318f1dab3b447ab09eba5eab344867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:11 UTC) from {username='slave'} at 127.0.0.1:41256
I20260926 13:44:11.085830 20350 tablet_replica.cc:330] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0: stopping tablet replica
I20260926 13:44:11.086148 20352 tablet_replica.cc:330] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27: stopping tablet replica
I20260926 13:44:11.086483 20352 raft_consensus.cc:2229] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:44:11.086671 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 19919
I20260926 13:44:11.086885 20352 raft_consensus.cc:2258] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:11.088789 20350 raft_consensus.cc:2229] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:44:11.089159 20350 raft_consensus.cc:2258] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:11.090477 20350 ts_tablet_manager.cc:1905] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:11.090584 20352 ts_tablet_manager.cc:1905] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:11.092991 20350 ts_tablet_manager.cc:1918] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:44:11.093113 20350 log.cc:1196] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/17318f1dab3b447ab09eba5eab344867
I20260926 13:44:11.093407 20350 ts_tablet_manager.cc:1939] T 17318f1dab3b447ab09eba5eab344867 P f25a39a5853c463480357102102b85a0: Deleting consensus metadata
I20260926 13:44:11.094167 20352 ts_tablet_manager.cc:1918] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:44:11.094277 20352 log.cc:1196] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/17318f1dab3b447ab09eba5eab344867
I20260926 13:44:11.094590 20352 ts_tablet_manager.cc:1939] T 17318f1dab3b447ab09eba5eab344867 P 025dfdeef84047cbbff6e4d28a4e0b27: Deleting consensus metadata
I20260926 13:44:11.094774 19843 catalog_manager.cc:4841] TS f25a39a5853c463480357102102b85a0 (127.1.49.2:37175): tablet 17318f1dab3b447ab09eba5eab344867 (table test-table [id=d2cc6a112b5f48c68da69d3330d37832]) successfully deleted
I20260926 13:44:11.095068 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 20050
W20260926 13:44:11.095129 19845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:45549 (error 108)
I20260926 13:44:11.096623 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:11.096231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:45549 (local address 127.0.0.1:57684)
0926 13:44:11.096438 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:11.096518 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:45549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 13:44:11.096750 19844 catalog_manager.cc:4841] TS 025dfdeef84047cbbff6e4d28a4e0b27 (127.1.49.3:43965): tablet 17318f1dab3b447ab09eba5eab344867 (table test-table [id=d2cc6a112b5f48c68da69d3330d37832]) successfully deleted
W20260926 13:44:11.096817 19845 catalog_manager.cc:4584] TS 70bba6b2b7e34a338c1b1d573e5d7012 (127.1.49.1:45549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17318f1dab3b447ab09eba5eab344867: Network error: Client connection negotiation failed: client connection to 127.1.49.1:45549: connect: Connection refused (error 111)
I20260926 13:44:11.101441 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 20181
I20260926 13:44:11.107338 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 19828
2026-09-26T13:44:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1142 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:11 dist-test-slave-1wns krb5kdc[20356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:11 dist-test-slave-1wns krb5kdc[20356](info): set up 2 sockets
Sep 26 13:44:11 dist-test-slave-1wns krb5kdc[20356](info): commencing operation
krb5kdc: starting...
W20260926 13:44:12.176071 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:12 dist-test-slave-1wns krb5kdc[20356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:44:12.315933 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:33645
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:40029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:33645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:12.445760 20369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:12.446125 20369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:12.446199 20369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:12.450917 20369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:12.451025 20369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:12.451061 20369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:12.451103 20369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:12.451125 20369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:12.457101 20369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:40029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:33645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:33645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:12.458333 20369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:12.459442 20369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:12.469082 20375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:12.469082 20374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:12.469075 20377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:12.469466 20369 server_base.cc:1034] running on GCE node
I20260926 13:44:12.470113 20369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:12.471424 20369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:12.472645 20369 hybrid_clock.cc:648] HybridClock initialized: now 1790430252472625 us; error 38 us; skew 500 ppm
Sep 26 13:44:12 dist-test-slave-1wns krb5kdc[20356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430252, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:12.475934 20369 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:44:12.477108 20369 webserver.cc:460] Webserver started at http://127.1.49.62:46039/ using document root <none> and password file <none>
I20260926 13:44:12.477744 20369 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:12.477842 20369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:12.478093 20369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:12.480520 20369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "a92a312c3000471c86929bc54b45e535"
format_stamp: "Formatted at 2026-09-26 13:44:12 on dist-test-slave-1wns"
server_key: "414271a8399cf027af93f6fcf6bf1b81"
server_key_iv: "2649f91b4ed630cf3da5d75631c5b018"
server_key_version: "encryptionkey@0"
I20260926 13:44:12.481175 20369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "a92a312c3000471c86929bc54b45e535"
format_stamp: "Formatted at 2026-09-26 13:44:12 on dist-test-slave-1wns"
server_key: "414271a8399cf027af93f6fcf6bf1b81"
server_key_iv: "2649f91b4ed630cf3da5d75631c5b018"
server_key_version: "encryptionkey@0"
I20260926 13:44:12.485586 20369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:44:12.488430 20384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:12.489931 20369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:12.490098 20369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "a92a312c3000471c86929bc54b45e535"
format_stamp: "Formatted at 2026-09-26 13:44:12 on dist-test-slave-1wns"
server_key: "414271a8399cf027af93f6fcf6bf1b81"
server_key_iv: "2649f91b4ed630cf3da5d75631c5b018"
server_key_version: "encryptionkey@0"
I20260926 13:44:12.490242 20369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:12.518899 20369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:12.522324 20369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:12.522550 20369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:12.530169 20369 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:33645
I20260926 13:44:12.530187 20436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:33645 every 8 connection(s)
I20260926 13:44:12.531278 20369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:12.533583 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 20369
I20260926 13:44:12.533738 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:12.534031 1220 external_mini_cluster.cc:1428] Setting key 6b685b8213b6da0d85b9dcd6dc9531ab
I20260926 13:44:12.534405 20437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:12.541721 20437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535: Bootstrap starting.
Sep 26 13:44:12 dist-test-slave-1wns krb5kdc[20356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:12.544341 20437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:12.545105 20437 log.cc:824] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:12.547658 20437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535: No bootstrap required, opened a new log
I20260926 13:44:12.550603 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:12.535464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.1.49.62:33645)
0926 13:44:12.535918 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:12.535935 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:12.535974 (+ 39us) server_negotiation.cc:407] Connection header received
0926 13:44:12.536683 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:12.536701 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:12.537051 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:12.537341 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:12.538273 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:12.539272 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:12.540520 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:12.541129 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:12.541294 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:12.544455 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:12.544464 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:12.544485 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:12.544528 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:12.546893 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:12.547407 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:12.547414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:12.547421 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:12.547573 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:12.547965 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:12.547969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:12.547971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:12.548558 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:12.548767 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:12.549090 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 13:44:12.549422 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":148,"threads_started":1}
I20260926 13:44:12.552814 20437 raft_consensus.cc:348] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92a312c3000471c86929bc54b45e535" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33645 } }
I20260926 13:44:12.553128 20437 raft_consensus.cc:374] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:12.553210 20437 raft_consensus.cc:729] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a92a312c3000471c86929bc54b45e535, State: Initialized, Role: FOLLOWER
I20260926 13:44:12.553609 20437 consensus_queue.cc:260] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92a312c3000471c86929bc54b45e535" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33645 } }
I20260926 13:44:12.553732 20437 raft_consensus.cc:388] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:12.553781 20437 raft_consensus.cc:482] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:12.553860 20437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:12.555025 20437 raft_consensus.cc:504] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92a312c3000471c86929bc54b45e535" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33645 } }
I20260926 13:44:12.555369 20437 leader_election.cc:302] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a92a312c3000471c86929bc54b45e535; no voters:
I20260926 13:44:12.555653 20437 leader_election.cc:288] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:12.555874 20442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:12.556202 20442 raft_consensus.cc:686] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [term 1 LEADER]: Becoming Leader. State: Replica: a92a312c3000471c86929bc54b45e535, State: Running, Role: LEADER
I20260926 13:44:12.556511 20442 consensus_queue.cc:237] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92a312c3000471c86929bc54b45e535" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33645 } }
I20260926 13:44:12.557089 20437 sys_catalog.cc:564] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:12.558358 20444 sys_catalog.cc:455] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a92a312c3000471c86929bc54b45e535. Latest consensus state: current_term: 1 leader_uuid: "a92a312c3000471c86929bc54b45e535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92a312c3000471c86929bc54b45e535" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33645 } } }
I20260926 13:44:12.558533 20444 sys_catalog.cc:458] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:12.558380 20443 sys_catalog.cc:455] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a92a312c3000471c86929bc54b45e535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92a312c3000471c86929bc54b45e535" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 33645 } } }
I20260926 13:44:12.558776 20443 sys_catalog.cc:458] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535 [sys.catalog]: This master's current role is: LEADER
W20260926 13:44:12.562716 20458 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:12.562806 20458 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:12.562889 20451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:12.563759 20451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:12.569232 20451 catalog_manager.cc:1347] Generated new cluster ID: 9eb31735275e44ba87fc75901cb905e2
I20260926 13:44:12.569331 20451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:12.578584 20451 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:12.579712 20451 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:12.599218 20451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535: Generated new TSK 0
I20260926 13:44:12.600271 20451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.49.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
Entry for principal kudu/127.1.49.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab.
I20260926 13:44:12.644104 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33645
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:40029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:12.767640 20463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:12.767967 20463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:12.768014 20463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:12.772303 20463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:12.772375 20463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:12.772472 20463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:44:12.778070 20463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:40029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.1.keytab
--principal=kudu/127.1.49.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:12.779242 20463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:12.780350 20463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:12.790684 20471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:12.790720 20469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:12.790798 20468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:12.790874 20463 server_base.cc:1034] running on GCE node
I20260926 13:44:12.791392 20463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:12.792128 20463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:12.793334 20463 hybrid_clock.cc:648] HybridClock initialized: now 1790430252793312 us; error 34 us; skew 500 ppm
Sep 26 13:44:12 dist-test-slave-1wns krb5kdc[20356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430252, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:12.796628 20463 init.cc:377] Logged in from keytab as kudu/127.1.49.1@KRBTEST.COM (short username kudu)
I20260926 13:44:12.797582 20463 webserver.cc:460] Webserver started at http://127.1.49.1:33705/ using document root <none> and password file <none>
I20260926 13:44:12.798161 20463 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:12.798224 20463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:12.798410 20463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:12.800765 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8"
format_stamp: "Formatted at 2026-09-26 13:44:12 on dist-test-slave-1wns"
server_key: "2d400989368740d08d51d20a959a0d28"
server_key_iv: "5d8363e1eabaa124a849df6d256fd559"
server_key_version: "encryptionkey@0"
I20260926 13:44:12.801487 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8"
format_stamp: "Formatted at 2026-09-26 13:44:12 on dist-test-slave-1wns"
server_key: "2d400989368740d08d51d20a959a0d28"
server_key_iv: "5d8363e1eabaa124a849df6d256fd559"
server_key_version: "encryptionkey@0"
I20260926 13:44:12.806291 20463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 13:44:12.809026 20478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:12.810530 20463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:12.810689 20463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8"
format_stamp: "Formatted at 2026-09-26 13:44:12 on dist-test-slave-1wns"
server_key: "2d400989368740d08d51d20a959a0d28"
server_key_iv: "5d8363e1eabaa124a849df6d256fd559"
server_key_version: "encryptionkey@0"
I20260926 13:44:12.810817 20463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:12.837384 20463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:12.840911 20463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:12.841131 20463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:12.841781 20463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:12.842667 20463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:12.842736 20463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:12.842818 20463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:12.842859 20463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:12.851728 20463 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:34615
I20260926 13:44:12.851791 20591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:34615 every 8 connection(s)
I20260926 13:44:12.852736 20463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:44:12.862327 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 20463
I20260926 13:44:12.862473 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:44:12.862773 1220 external_mini_cluster.cc:1428] Setting key 076a23a31cad6afaa77bf820bfb02702
Sep 26 13:44:12 dist-test-slave-1wns krb5kdc[20356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430252, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.1@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:12.868064 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:12.854698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:38709 (local address 127.1.49.62:33645)
0926 13:44:12.854855 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:12.854859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:12.855906 (+ 1047us) server_negotiation.cc:407] Connection header received
0926 13:44:12.856871 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:12.856875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:12.856939 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:12.857037 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:12.858476 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:12.859022 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:12.860324 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:12.860603 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:12.860720 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:12.863626 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:12.863630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:12.863633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:12.863665 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:12.865470 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:12.866092 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:12.866098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:12.866103 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:12.866209 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:12.866594 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:12.866599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:12.866601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:12.866888 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:12.867004 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:12.867603 (+ 599us) server_negotiation.cc:299] Negotiation successful
0926 13:44:12.867776 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 13:44:12.869060 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:12.855099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33645 (local address 127.1.49.1:38709)
0926 13:44:12.855642 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:12.855690 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:12.856637 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:12.857229 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:12.857238 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:12.857676 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:12.858334 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:12.858350 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:12.859151 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:12.859155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:12.860196 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:12.860206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:12.860767 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:12.860772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:12.860944 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:12.861553 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:12.861570 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:12.863458 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:12.865590 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:12.865596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:12.865610 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:12.865953 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:12.866340 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:12.866343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:12.866351 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:12.866496 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:12.867063 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:12.867069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:12.867349 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 13:44:12.867528 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 13:44:12.867904 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":138,"threads_started":1}
I20260926 13:44:12.870533 20592 heartbeater.cc:347] Connected to a master server at 127.1.49.62:33645
I20260926 13:44:12.870875 20592 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:12.871433 20592 heartbeater.cc:511] Master 127.1.49.62:33645 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.49.2@KRBTEST.COM; defaulting to no policy
I20260926 13:44:12.873421 20401 ts_manager.cc:194] Registered new tserver with Master: 8b8f684aa58e4dbeb3bc6f7d239da3f8 (127.1.49.1:34615)
I20260926 13:44:12.874603 20401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.1@KRBTEST.COM'} at 127.1.49.1:38709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
Entry for principal kudu/127.1.49.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab.
I20260926 13:44:12.895403 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33645
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:40029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:13.013844 20597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:13.014182 20597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:13.014287 20597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:13.018757 20597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:13.018846 20597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:13.018910 20597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:44:13.024648 20597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:40029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.2.keytab
--principal=kudu/127.1.49.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:13.025873 20597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:13.026995 20597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:13.037906 20603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:13.037907 20605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:13.037920 20602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:13.038239 20597 server_base.cc:1034] running on GCE node
I20260926 13:44:13.038793 20597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:13.039541 20597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:13.040721 20597 hybrid_clock.cc:648] HybridClock initialized: now 1790430253040652 us; error 80 us; skew 500 ppm
Sep 26 13:44:13 dist-test-slave-1wns krb5kdc[20356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430253, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:13.044023 20597 init.cc:377] Logged in from keytab as kudu/127.1.49.2@KRBTEST.COM (short username kudu)
I20260926 13:44:13.045077 20597 webserver.cc:460] Webserver started at http://127.1.49.2:38567/ using document root <none> and password file <none>
I20260926 13:44:13.045702 20597 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:13.045782 20597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:13.046006 20597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:13.048334 20597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "4deb368655ee413cad9f44dac6be6a05"
format_stamp: "Formatted at 2026-09-26 13:44:13 on dist-test-slave-1wns"
server_key: "a2d278d3a7ffb2cfa610e74a876905d8"
server_key_iv: "9999d5bea144e30afe2666daa7e778b2"
server_key_version: "encryptionkey@0"
I20260926 13:44:13.049000 20597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "4deb368655ee413cad9f44dac6be6a05"
format_stamp: "Formatted at 2026-09-26 13:44:13 on dist-test-slave-1wns"
server_key: "a2d278d3a7ffb2cfa610e74a876905d8"
server_key_iv: "9999d5bea144e30afe2666daa7e778b2"
server_key_version: "encryptionkey@0"
I20260926 13:44:13.053324 20597 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:44:13.056290 20612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:13.058144 20597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 13:44:13.058302 20597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "4deb368655ee413cad9f44dac6be6a05"
format_stamp: "Formatted at 2026-09-26 13:44:13 on dist-test-slave-1wns"
server_key: "a2d278d3a7ffb2cfa610e74a876905d8"
server_key_iv: "9999d5bea144e30afe2666daa7e778b2"
server_key_version: "encryptionkey@0"
I20260926 13:44:13.058470 20597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:13.088043 20597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:13.091907 20597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:13.092175 20597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:13.092881 20597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:13.093953 20597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:13.094024 20597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:13.094106 20597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:13.094151 20597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:13.104979 20597 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:46041
I20260926 13:44:13.105036 20725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:46041 every 8 connection(s)
I20260926 13:44:13.106076 20597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:44:13.113406 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 20597
I20260926 13:44:13.113550 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:44:13.113873 1220 external_mini_cluster.cc:1428] Setting key 88f852f98dd598e58c3acd60ad432ff2
Sep 26 13:44:13 dist-test-slave-1wns krb5kdc[20356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430253, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.2@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:13.122825 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.108245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:55817 (local address 127.1.49.62:33645)
0926 13:44:13.108423 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:13.108426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:13.109492 (+ 1066us) server_negotiation.cc:407] Connection header received
0926 13:44:13.110474 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:13.110477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:13.110531 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:13.110615 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:13.111990 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.112552 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.114216 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.114595 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.114808 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.117943 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:13.117962 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:13.117964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:13.117994 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:13.120046 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:13.120784 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:13.120791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:13.120796 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:13.120907 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:13.121362 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:13.121367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:13.121372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:13.121674 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:13.121813 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:13.122451 (+ 638us) server_negotiation.cc:299] Negotiation successful
0926 13:44:13.122606 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 13:44:13.123689 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.108713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33645 (local address 127.1.49.2:55817)
0926 13:44:13.109295 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:13.109356 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:13.110239 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:13.110802 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:13.110809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:13.111237 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:13.111770 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:13.111786 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.112719 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.112722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:13.114067 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:13.114080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.114847 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.114850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:13.115009 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.115743 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:13.115765 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:13.117724 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:13.120255 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:13.120265 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:13.120294 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:13.120651 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:13.121047 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:13.121055 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:13.121057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:13.121238 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:13.121859 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:13.121865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:13.122169 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 13:44:13.122354 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 13:44:13.122667 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":133,"threads_started":1}
WARNING: no policy specified for kudu/127.1.49.3@KRBTEST.COM; defaulting to no policy
I20260926 13:44:13.125077 20726 heartbeater.cc:347] Connected to a master server at 127.1.49.62:33645
I20260926 13:44:13.125337 20726 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:13.125896 20726 heartbeater.cc:511] Master 127.1.49.62:33645 requested a full tablet report, sending...
I20260926 13:44:13.127260 20401 ts_manager.cc:194] Registered new tserver with Master: 4deb368655ee413cad9f44dac6be6a05 (127.1.49.2:46041)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.3@KRBTEST.COM" created.
I20260926 13:44:13.128270 20401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.2@KRBTEST.COM'} at 127.1.49.2:55817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
Entry for principal kudu/127.1.49.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab.
I20260926 13:44:13.148321 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33645
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:40029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:13.264683 20731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:13.264999 20731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:13.265096 20731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:13.269481 20731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:13.269577 20731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:13.269702 20731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:44:13.275395 20731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:40029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.3.keytab
--principal=kudu/127.1.49.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:33645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:13.276595 20731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:13.277679 20731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:13.287446 20737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:13.287433 20739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:13.287442 20736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:13.287547 20731 server_base.cc:1034] running on GCE node
I20260926 13:44:13.288254 20731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:13.288920 20731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:13.290112 20731 hybrid_clock.cc:648] HybridClock initialized: now 1790430253290082 us; error 45 us; skew 500 ppm
Sep 26 13:44:13 dist-test-slave-1wns krb5kdc[20356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430253, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:13.293166 20731 init.cc:377] Logged in from keytab as kudu/127.1.49.3@KRBTEST.COM (short username kudu)
I20260926 13:44:13.294428 20731 webserver.cc:460] Webserver started at http://127.1.49.3:42599/ using document root <none> and password file <none>
I20260926 13:44:13.295156 20731 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:13.295256 20731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:13.295476 20731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:13.298560 20731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "09301a78f8f340fa99bb5a6a5fe1fb40"
format_stamp: "Formatted at 2026-09-26 13:44:13 on dist-test-slave-1wns"
server_key: "6430ab79d25fd5d59162fb4eff3f1180"
server_key_iv: "8713ff84d31fb85914a633ccc607f95c"
server_key_version: "encryptionkey@0"
I20260926 13:44:13.299285 20731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "09301a78f8f340fa99bb5a6a5fe1fb40"
format_stamp: "Formatted at 2026-09-26 13:44:13 on dist-test-slave-1wns"
server_key: "6430ab79d25fd5d59162fb4eff3f1180"
server_key_iv: "8713ff84d31fb85914a633ccc607f95c"
server_key_version: "encryptionkey@0"
I20260926 13:44:13.303531 20731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 13:44:13.306422 20746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:13.307776 20731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:44:13.307988 20731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "09301a78f8f340fa99bb5a6a5fe1fb40"
format_stamp: "Formatted at 2026-09-26 13:44:13 on dist-test-slave-1wns"
server_key: "6430ab79d25fd5d59162fb4eff3f1180"
server_key_iv: "8713ff84d31fb85914a633ccc607f95c"
server_key_version: "encryptionkey@0"
I20260926 13:44:13.308105 20731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:13.335765 20731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:13.339552 20731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:13.339789 20731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:13.340473 20731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:13.341352 20731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:13.341401 20731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:13.341466 20731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:13.341499 20731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:13.350996 20731 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:40677
I20260926 13:44:13.351020 20859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:40677 every 8 connection(s)
I20260926 13:44:13.351985 20731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:44:13.356104 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 20731
I20260926 13:44:13.356262 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:44:13.356621 1220 external_mini_cluster.cc:1428] Setting key 4e1a8153f875ffffbb48d164d5153baa
Sep 26 13:44:13 dist-test-slave-1wns krb5kdc[20356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430253, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.3@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:13.367924 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.354164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:35057 (local address 127.1.49.62:33645)
0926 13:44:13.354310 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:13.354314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:13.355095 (+ 781us) server_negotiation.cc:407] Connection header received
0926 13:44:13.356642 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:13.356647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:13.356714 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:13.356827 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:13.358652 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.359484 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.360881 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.361134 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.361234 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.364243 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:13.364248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:13.364250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:13.364283 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:13.365637 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:13.366222 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:13.366226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:13.366227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:13.366337 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:13.366674 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:13.366677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:13.366678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:13.366961 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:13.367045 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:13.367564 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 13:44:13.367674 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 13:44:13.368770 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.354380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:33645 (local address 127.1.49.3:35057)
0926 13:44:13.354929 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:13.354978 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:13.356402 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:13.357055 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:13.357064 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:13.357658 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:13.358497 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:13.358522 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.359628 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.359631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:13.360701 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:13.360712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.361313 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.361316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:13.361489 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.362169 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:13.362186 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:13.364061 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:13.365742 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:13.365747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:13.365761 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:13.366076 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:13.366427 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:13.366429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:13.366430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:13.366562 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:13.367051 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:13.367056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:13.367301 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 13:44:13.367507 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 13:44:13.367798 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
I20260926 13:44:13.370268 20860 heartbeater.cc:347] Connected to a master server at 127.1.49.62:33645
I20260926 13:44:13.370577 20860 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:13.371212 20860 heartbeater.cc:511] Master 127.1.49.62:33645 requested a full tablet report, sending...
I20260926 13:44:13.372303 20401 ts_manager.cc:194] Registered new tserver with Master: 09301a78f8f340fa99bb5a6a5fe1fb40 (127.1.49.3:40677)
I20260926 13:44:13.372939 20401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.49.3@KRBTEST.COM'} at 127.1.49.3:35057
I20260926 13:44:13.383409 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:44:13.394245 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.385588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.1.49.62:33645)
0926 13:44:13.385920 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:13.385926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:13.386028 (+ 102us) server_negotiation.cc:407] Connection header received
0926 13:44:13.386194 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:13.386196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:13.386239 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:13.386338 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:13.387100 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.387660 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.388589 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.388820 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.388932 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.390567 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:13.390672 (+ 105us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:13.390676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:13.390708 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:13.392413 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:13.392856 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:13.392860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:13.392862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:13.392945 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:13.393228 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:13.393232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:13.393234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:13.393512 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:13.393652 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:13.393955 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 13:44:13.394078 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260926 13:44:13.397519 20401 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:44:13.400672 20401 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:44:13.416518 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.409683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.1.49.2:46041)
0926 13:44:13.410072 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:13.410084 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:13.410466 (+ 382us) server_negotiation.cc:407] Connection header received
0926 13:44:13.410782 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:13.410790 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:13.411027 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:13.411164 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:13.411715 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.413031 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.414845 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.415736 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.415918 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.416096 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:13.416233 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 13:44:13.416344 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":92,"threads_started":1}
I20260926 13:44:13.416512 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.409884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:34615 (local address 127.0.0.1:38800)
0926 13:44:13.410521 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:13.410534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:13.410599 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:13.412094 (+ 1495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:13.412097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:13.412115 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:13.412208 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:13.412218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.413302 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.413305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:13.414893 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:13.414904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.415988 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.415991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:13.416077 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.416099 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:44:13.416174 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 13:44:13.416346 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":217,"thread_start_us":78,"threads_started":1}
I20260926 13:44:13.416924 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.410387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.1.49.3:40677)
0926 13:44:13.410991 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:13.411005 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:13.411029 (+ 24us) server_negotiation.cc:407] Connection header received
0926 13:44:13.411105 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:13.411110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:13.411302 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:13.411473 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:13.411828 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.413031 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.415151 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.415875 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.415980 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.416120 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:13.416584 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 13:44:13.416764 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":109,"threads_started":1}
I20260926 13:44:13.417111 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.409558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:46041 (local address 127.0.0.1:49350)
0926 13:44:13.410311 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:13.410372 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:13.410583 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:13.411345 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:13.411354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:13.411382 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:13.411546 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:13.411554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.413200 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.413204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:13.414723 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:13.414734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.415905 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.415912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:13.416028 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.416066 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 13:44:13.416190 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 13:44:13.416972 (+ 782us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":156,"thread_start_us":90,"threads_started":1}
I20260926 13:44:13.416512 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.409632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:40677 (local address 127.0.0.1:44276)
0926 13:44:13.410482 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:13.410497 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:13.410595 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:13.411503 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:13.411506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:13.411526 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:13.411643 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:13.411651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.413200 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.413204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:13.415035 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:13.415042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.416181 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.416186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:13.416267 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.416292 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 13:44:13.416367 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 13:44:13.416424 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":246,"thread_start_us":110,"threads_started":1}
I20260926 13:44:13.417543 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.410773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.1.49.1:34615)
0926 13:44:13.411638 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:13.411644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:13.411667 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:44:13.411745 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:13.411753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:13.411955 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:13.412104 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:13.412301 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.413171 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.415019 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.415860 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.416327 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.416464 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:13.416512 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 13:44:13.417152 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":141,"threads_started":1}
I20260926 13:44:13.420192 20660 tablet_service.cc:1467] Processing CreateTablet for tablet 62d6e2eacbea4aa5ba68453af2f5cbf1 (DEFAULT_TABLE table=test-table [id=f0f7ca6d9a2b4278ae06aded0f422882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:13.420337 20526 tablet_service.cc:1467] Processing CreateTablet for tablet 62d6e2eacbea4aa5ba68453af2f5cbf1 (DEFAULT_TABLE table=test-table [id=f0f7ca6d9a2b4278ae06aded0f422882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:13.420428 20794 tablet_service.cc:1467] Processing CreateTablet for tablet 62d6e2eacbea4aa5ba68453af2f5cbf1 (DEFAULT_TABLE table=test-table [id=f0f7ca6d9a2b4278ae06aded0f422882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:13.421365 20526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d6e2eacbea4aa5ba68453af2f5cbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:13.421365 20794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d6e2eacbea4aa5ba68453af2f5cbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:13.421370 20660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d6e2eacbea4aa5ba68453af2f5cbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:13.428027 20877 tablet_bootstrap.cc:492] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05: Bootstrap starting.
I20260926 13:44:13.429383 20878 tablet_bootstrap.cc:492] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8: Bootstrap starting.
I20260926 13:44:13.430997 20877 tablet_bootstrap.cc:654] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:13.431214 20878 tablet_bootstrap.cc:654] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:13.431931 20877 log.cc:824] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:13.431990 20878 log.cc:824] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:13.432307 20879 tablet_bootstrap.cc:492] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40: Bootstrap starting.
I20260926 13:44:13.433938 20878 tablet_bootstrap.cc:492] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8: No bootstrap required, opened a new log
I20260926 13:44:13.434134 20878 ts_tablet_manager.cc:1397] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 13:44:13.434316 20879 tablet_bootstrap.cc:654] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:13.435379 20879 log.cc:824] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:13.436249 20877 tablet_bootstrap.cc:492] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05: No bootstrap required, opened a new log
I20260926 13:44:13.436470 20877 ts_tablet_manager.cc:1397] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 13:44:13.437112 20879 tablet_bootstrap.cc:492] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40: No bootstrap required, opened a new log
I20260926 13:44:13.437304 20879 ts_tablet_manager.cc:1397] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:44:13.439023 20878 raft_consensus.cc:348] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deb368655ee413cad9f44dac6be6a05" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46041 } } peers { permanent_uuid: "09301a78f8f340fa99bb5a6a5fe1fb40" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40677 } } peers { permanent_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34615 } }
I20260926 13:44:13.439355 20878 raft_consensus.cc:374] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:13.439422 20878 raft_consensus.cc:729] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8f684aa58e4dbeb3bc6f7d239da3f8, State: Initialized, Role: FOLLOWER
I20260926 13:44:13.439967 20878 consensus_queue.cc:260] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deb368655ee413cad9f44dac6be6a05" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46041 } } peers { permanent_uuid: "09301a78f8f340fa99bb5a6a5fe1fb40" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40677 } } peers { permanent_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34615 } }
I20260926 13:44:13.440446 20877 raft_consensus.cc:348] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deb368655ee413cad9f44dac6be6a05" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46041 } } peers { permanent_uuid: "09301a78f8f340fa99bb5a6a5fe1fb40" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40677 } } peers { permanent_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34615 } }
I20260926 13:44:13.440729 20877 raft_consensus.cc:374] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:13.440752 20592 heartbeater.cc:503] Master 127.1.49.62:33645 was elected leader, sending a full tablet report...
I20260926 13:44:13.440779 20877 raft_consensus.cc:729] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4deb368655ee413cad9f44dac6be6a05, State: Initialized, Role: FOLLOWER
I20260926 13:44:13.441116 20878 ts_tablet_manager.cc:1428] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 13:44:13.441059 20879 raft_consensus.cc:348] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deb368655ee413cad9f44dac6be6a05" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46041 } } peers { permanent_uuid: "09301a78f8f340fa99bb5a6a5fe1fb40" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40677 } } peers { permanent_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34615 } }
I20260926 13:44:13.441309 20879 raft_consensus.cc:374] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:13.441236 20877 consensus_queue.cc:260] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deb368655ee413cad9f44dac6be6a05" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46041 } } peers { permanent_uuid: "09301a78f8f340fa99bb5a6a5fe1fb40" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40677 } } peers { permanent_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34615 } }
I20260926 13:44:13.441359 20879 raft_consensus.cc:729] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09301a78f8f340fa99bb5a6a5fe1fb40, State: Initialized, Role: FOLLOWER
I20260926 13:44:13.441776 20879 consensus_queue.cc:260] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deb368655ee413cad9f44dac6be6a05" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46041 } } peers { permanent_uuid: "09301a78f8f340fa99bb5a6a5fe1fb40" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40677 } } peers { permanent_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34615 } }
I20260926 13:44:13.442081 20726 heartbeater.cc:503] Master 127.1.49.62:33645 was elected leader, sending a full tablet report...
I20260926 13:44:13.442241 20877 ts_tablet_manager.cc:1428] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:44:13.442698 20879 ts_tablet_manager.cc:1428] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260926 13:44:13.443055 20860 heartbeater.cc:503] Master 127.1.49.62:33645 was elected leader, sending a full tablet report...
I20260926 13:44:13.546487 20883 raft_consensus.cc:482] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:44:13.546730 20883 raft_consensus.cc:504] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deb368655ee413cad9f44dac6be6a05" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46041 } } peers { permanent_uuid: "09301a78f8f340fa99bb5a6a5fe1fb40" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40677 } } peers { permanent_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34615 } }
I20260926 13:44:13.548108 20883 leader_election.cc:288] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4deb368655ee413cad9f44dac6be6a05 (127.1.49.2:46041), 09301a78f8f340fa99bb5a6a5fe1fb40 (127.1.49.3:40677)
I20260926 13:44:13.554150 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.548497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:34445 (local address 127.1.49.2:46041)
0926 13:44:13.548670 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:13.548674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:13.548953 (+ 279us) server_negotiation.cc:407] Connection header received
0926 13:44:13.549198 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:13.549205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:13.549270 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:13.549325 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:13.549914 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.550716 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.552773 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.553420 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.553535 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.553602 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:13.553936 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 13:44:13.554012 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 13:44:13.554150 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.548589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:54541 (local address 127.1.49.3:40677)
0926 13:44:13.548723 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:13.548728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:13.548983 (+ 255us) server_negotiation.cc:407] Connection header received
0926 13:44:13.549196 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:13.549258 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:13.549348 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:13.549430 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:13.549861 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.550716 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.552773 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.553421 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.553535 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.553602 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:13.553929 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 13:44:13.554012 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:44:13.554257 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.548467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:40677 (local address 127.1.49.1:54541)
0926 13:44:13.548872 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:13.548884 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:13.549070 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:13.549466 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:13.549469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:13.549485 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:13.549737 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:13.549748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.550853 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.550927 (+ 74us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:13.552624 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:13.552636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.553560 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.553564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:13.553714 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.553788 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 13:44:13.553879 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 13:44:13.554054 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":374,"thread_start_us":128,"threads_started":1}
I20260926 13:44:13.554266 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.548379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:46041 (local address 127.1.49.1:34445)
0926 13:44:13.548820 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:13.548833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:13.549041 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:13.549491 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:13.549496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:13.549516 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:13.549744 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:13.549753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.550853 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.550857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:13.552624 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:13.552636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.553560 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.553564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:13.553729 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.553791 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 13:44:13.553874 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 13:44:13.554054 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":258,"spinlock_wait_cycles":9600,"thread_start_us":155,"threads_started":1}
I20260926 13:44:13.555037 20814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d6e2eacbea4aa5ba68453af2f5cbf1" candidate_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09301a78f8f340fa99bb5a6a5fe1fb40" is_pre_election: true
I20260926 13:44:13.555063 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d6e2eacbea4aa5ba68453af2f5cbf1" candidate_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4deb368655ee413cad9f44dac6be6a05" is_pre_election: true
I20260926 13:44:13.555349 20814 raft_consensus.cc:2454] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8f684aa58e4dbeb3bc6f7d239da3f8 in term 0.
I20260926 13:44:13.555352 20680 raft_consensus.cc:2454] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8f684aa58e4dbeb3bc6f7d239da3f8 in term 0.
I20260926 13:44:13.555905 20481 leader_election.cc:302] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09301a78f8f340fa99bb5a6a5fe1fb40, 4deb368655ee413cad9f44dac6be6a05, 8b8f684aa58e4dbeb3bc6f7d239da3f8; no voters:
I20260926 13:44:13.556169 20883 raft_consensus.cc:2781] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:44:13.556263 20883 raft_consensus.cc:482] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:44:13.556305 20883 raft_consensus.cc:3037] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:13.557475 20883 raft_consensus.cc:504] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deb368655ee413cad9f44dac6be6a05" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46041 } } peers { permanent_uuid: "09301a78f8f340fa99bb5a6a5fe1fb40" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40677 } } peers { permanent_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34615 } }
I20260926 13:44:13.557835 20883 leader_election.cc:288] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [CANDIDATE]: Term 1 election: Requested vote from peers 4deb368655ee413cad9f44dac6be6a05 (127.1.49.2:46041), 09301a78f8f340fa99bb5a6a5fe1fb40 (127.1.49.3:40677)
I20260926 13:44:13.558156 20680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d6e2eacbea4aa5ba68453af2f5cbf1" candidate_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4deb368655ee413cad9f44dac6be6a05"
I20260926 13:44:13.558238 20814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d6e2eacbea4aa5ba68453af2f5cbf1" candidate_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09301a78f8f340fa99bb5a6a5fe1fb40"
I20260926 13:44:13.558301 20680 raft_consensus.cc:3037] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:13.558384 20814 raft_consensus.cc:3037] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:13.559455 20680 raft_consensus.cc:2454] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8f684aa58e4dbeb3bc6f7d239da3f8 in term 1.
I20260926 13:44:13.559613 20814 raft_consensus.cc:2454] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8f684aa58e4dbeb3bc6f7d239da3f8 in term 1.
I20260926 13:44:13.559773 20481 leader_election.cc:302] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4deb368655ee413cad9f44dac6be6a05, 8b8f684aa58e4dbeb3bc6f7d239da3f8; no voters:
I20260926 13:44:13.559993 20883 raft_consensus.cc:2781] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:13.560241 20883 raft_consensus.cc:686] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 8b8f684aa58e4dbeb3bc6f7d239da3f8, State: Running, Role: LEADER
I20260926 13:44:13.560622 20883 consensus_queue.cc:237] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deb368655ee413cad9f44dac6be6a05" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46041 } } peers { permanent_uuid: "09301a78f8f340fa99bb5a6a5fe1fb40" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40677 } } peers { permanent_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34615 } }
I20260926 13:44:13.563969 20399 catalog_manager.cc:5495] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8f684aa58e4dbeb3bc6f7d239da3f8 (127.1.49.1). New cstate: current_term: 1 leader_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deb368655ee413cad9f44dac6be6a05" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09301a78f8f340fa99bb5a6a5fe1fb40" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8f684aa58e4dbeb3bc6f7d239da3f8" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34615 } health_report { overall_health: HEALTHY } } }
W20260926 13:44:13.602952 20861 tablet.cc:2369] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:44:13.604225 20593 tablet.cc:2369] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:44:13.607450 20727 tablet.cc:2369] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:44:13.607632 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.603398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.1.49.1:34615)
0926 13:44:13.603569 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:13.603573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:13.603592 (+ 19us) server_negotiation.cc:407] Connection header received
0926 13:44:13.603688 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:13.603691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:13.603747 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:13.603865 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:13.604192 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.604962 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.605923 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.606178 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.606286 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.606628 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:13.607167 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:13.607268 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:13.607397 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 13:44:13.607459 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 13:44:13.613801 20814 raft_consensus.cc:1261] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40 [term 1 FOLLOWER]: Refusing update from remote peer 8b8f684aa58e4dbeb3bc6f7d239da3f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:13.613862 20680 raft_consensus.cc:1261] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05 [term 1 FOLLOWER]: Refusing update from remote peer 8b8f684aa58e4dbeb3bc6f7d239da3f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:13.614610 20888 consensus_queue.cc:1035] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4deb368655ee413cad9f44dac6be6a05" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 46041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:13.614851 20883 consensus_queue.cc:1035] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 8b8f684aa58e4dbeb3bc6f7d239da3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09301a78f8f340fa99bb5a6a5fe1fb40" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 40677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:13.626593 20892 mvcc.cc:205] Tried to move back new op lower bound from 7333602318794510336 to 7333602318587088896. Current Snapshot: MvccSnapshot[applied={T|T < 7333602318794510336 or (T in {7333602318794510336})}]
I20260926 13:44:13.632663 20399 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:13.632899 20399 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:13.635895 20399 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a92a312c3000471c86929bc54b45e535: Sending DeleteTablet for 3 replicas of tablet 62d6e2eacbea4aa5ba68453af2f5cbf1
I20260926 13:44:13.636854 20660 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d6e2eacbea4aa5ba68453af2f5cbf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:13 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:49350
I20260926 13:44:13.636931 20794 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d6e2eacbea4aa5ba68453af2f5cbf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:13 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:44276
I20260926 13:44:13.637015 20526 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d6e2eacbea4aa5ba68453af2f5cbf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:13 UTC) from {username='kudu', principal='kudu/127.1.49.62@KRBTEST.COM'} at 127.0.0.1:38800
I20260926 13:44:13.637534 20901 tablet_replica.cc:330] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05: stopping tablet replica
I20260926 13:44:13.637599 20902 tablet_replica.cc:330] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40: stopping tablet replica
I20260926 13:44:13.637923 20901 raft_consensus.cc:2229] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:44:13.637939 20902 raft_consensus.cc:2229] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:44:13.638300 20902 raft_consensus.cc:2258] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:13.638309 20901 raft_consensus.cc:2258] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:13.638605 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 20463
I20260926 13:44:13.640209 20902 ts_tablet_manager.cc:1905] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:13.642500 20901 ts_tablet_manager.cc:1905] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:13.643400 20902 ts_tablet_manager.cc:1918] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:44:13.643507 20902 log.cc:1196] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/62d6e2eacbea4aa5ba68453af2f5cbf1
I20260926 13:44:13.643935 20902 ts_tablet_manager.cc:1939] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 09301a78f8f340fa99bb5a6a5fe1fb40: Deleting consensus metadata
I20260926 13:44:13.645438 20388 catalog_manager.cc:4841] TS 09301a78f8f340fa99bb5a6a5fe1fb40 (127.1.49.3:40677): tablet 62d6e2eacbea4aa5ba68453af2f5cbf1 (table test-table [id=f0f7ca6d9a2b4278ae06aded0f422882]) successfully deleted
I20260926 13:44:13.645558 20901 ts_tablet_manager.cc:1918] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:44:13.645654 20901 log.cc:1196] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790429983839632-1220-0/minicluster-data/ts-1/wal/wals/62d6e2eacbea4aa5ba68453af2f5cbf1
I20260926 13:44:13.646006 20901 ts_tablet_manager.cc:1939] T 62d6e2eacbea4aa5ba68453af2f5cbf1 P 4deb368655ee413cad9f44dac6be6a05: Deleting consensus metadata
W20260926 13:44:13.646543 20385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:34615 (error 108)
I20260926 13:44:13.647274 20387 catalog_manager.cc:4841] TS 4deb368655ee413cad9f44dac6be6a05 (127.1.49.2:46041): tablet 62d6e2eacbea4aa5ba68453af2f5cbf1 (table test-table [id=f0f7ca6d9a2b4278ae06aded0f422882]) successfully deleted
I20260926 13:44:13.647696 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.647329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:34615 (local address 127.0.0.1:38816)
0926 13:44:13.647512 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:13.647593 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 13:44:13.647794 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 20597
W20260926 13:44:13.647893 20385 catalog_manager.cc:4584] TS 8b8f684aa58e4dbeb3bc6f7d239da3f8 (127.1.49.1:34615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62d6e2eacbea4aa5ba68453af2f5cbf1: Network error: Client connection negotiation failed: client connection to 127.1.49.1:34615: connect: Connection refused (error 111)
I20260926 13:44:13.653517 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 20731
I20260926 13:44:13.659688 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 20369
2026-09-26T13:44:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2553 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T13:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:13Z Disabled control of system clock
I20260926 13:44:13.691200 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:37539
--webserver_interface=127.1.49.62
--webserver_port=0
--builtin_ntp_servers=127.1.49.20:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:37539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:44:13.810874 20908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:13.811326 20908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:13.811401 20908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:13.817510 20908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:13.817602 20908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:13.817628 20908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:13.817657 20908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:13.817692 20908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:13.826303 20908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:37539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:37539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:13.828001 20908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:13.829768 20908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:13.841250 20914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:13.841665 20908 server_base.cc:1034] running on GCE node
W20260926 13:44:13.846294 20916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:13.846320 20913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:13.847347 20908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:13.849305 20908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:13.850508 20908 hybrid_clock.cc:648] HybridClock initialized: now 1790430253850473 us; error 60 us; skew 500 ppm
I20260926 13:44:13.853461 20908 webserver.cc:460] Webserver started at http://127.1.49.62:33547/ using document root <none> and password file <none>
I20260926 13:44:13.854175 20908 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:13.854290 20908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:13.854569 20908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:13.857151 20908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "3e9adacf16d54c788cb0f9127eb14bce"
format_stamp: "Formatted at 2026-09-26 13:44:13 on dist-test-slave-1wns"
server_key: "2c670a108ed6a38930ed8f4724c087fb"
server_key_iv: "df94ab4c20753fef5689a131846cf341"
server_key_version: "encryptionkey@0"
I20260926 13:44:13.857815 20908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "3e9adacf16d54c788cb0f9127eb14bce"
format_stamp: "Formatted at 2026-09-26 13:44:13 on dist-test-slave-1wns"
server_key: "2c670a108ed6a38930ed8f4724c087fb"
server_key_iv: "df94ab4c20753fef5689a131846cf341"
server_key_version: "encryptionkey@0"
I20260926 13:44:13.862207 20908 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:44:13.865176 20922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:13.866578 20908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:44:13.866739 20908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "3e9adacf16d54c788cb0f9127eb14bce"
format_stamp: "Formatted at 2026-09-26 13:44:13 on dist-test-slave-1wns"
server_key: "2c670a108ed6a38930ed8f4724c087fb"
server_key_iv: "df94ab4c20753fef5689a131846cf341"
server_key_version: "encryptionkey@0"
I20260926 13:44:13.866876 20908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:13.880722 20908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:13.881494 20908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:13.881702 20908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:13.889694 20908 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:37539
I20260926 13:44:13.889712 20974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:37539 every 8 connection(s)
I20260926 13:44:13.891163 20908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:13.894587 20975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:13.897696 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 20908
I20260926 13:44:13.897900 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:13.898203 1220 external_mini_cluster.cc:1428] Setting key 064d203aa4fc89a31ac7a56d0eeaadd1
I20260926 13:44:13.900857 20975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce: Bootstrap starting.
I20260926 13:44:13.903364 20975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:13.904248 20975 log.cc:824] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:13.906663 20975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce: No bootstrap required, opened a new log
I20260926 13:44:13.908869 20978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:13.899909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.1.49.62:37539)
0926 13:44:13.900628 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:13.900647 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:13.900687 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:44:13.901540 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:13.901561 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:13.902022 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:13.902364 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:13.902780 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.904009 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.905101 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:13.905636 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:13.905807 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:13.906852 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:13.906860 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:13.906875 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:13.907322 (+ 447us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:13.907397 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:13.907420 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:13.907509 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:13.907676 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 13:44:13.908073 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":163,"threads_started":1}
I20260926 13:44:13.910581 20975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9adacf16d54c788cb0f9127eb14bce" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 37539 } }
I20260926 13:44:13.910801 20975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:13.910856 20975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e9adacf16d54c788cb0f9127eb14bce, State: Initialized, Role: FOLLOWER
I20260926 13:44:13.911340 20975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9adacf16d54c788cb0f9127eb14bce" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 37539 } }
I20260926 13:44:13.911497 20975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:13.911564 20975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:13.911687 20975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:13.912796 20975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9adacf16d54c788cb0f9127eb14bce" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 37539 } }
I20260926 13:44:13.913153 20975 leader_election.cc:302] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e9adacf16d54c788cb0f9127eb14bce; no voters:
I20260926 13:44:13.913502 20975 leader_election.cc:288] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:13.913654 20980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:13.913988 20980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [term 1 LEADER]: Becoming Leader. State: Replica: 3e9adacf16d54c788cb0f9127eb14bce, State: Running, Role: LEADER
I20260926 13:44:13.914342 20980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9adacf16d54c788cb0f9127eb14bce" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 37539 } }
I20260926 13:44:13.914856 20975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:13.916065 20982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e9adacf16d54c788cb0f9127eb14bce. Latest consensus state: current_term: 1 leader_uuid: "3e9adacf16d54c788cb0f9127eb14bce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9adacf16d54c788cb0f9127eb14bce" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 37539 } } }
I20260926 13:44:13.916209 20982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:13.916564 20981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e9adacf16d54c788cb0f9127eb14bce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9adacf16d54c788cb0f9127eb14bce" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 37539 } } }
I20260926 13:44:13.916663 20989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:13.916680 20981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:13.920116 20989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:13.926640 20989 catalog_manager.cc:1347] Generated new cluster ID: 7cefd5e8a1e64f2a9285315e82d7f26e
I20260926 13:44:13.926738 20989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:13.940483 20989 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:13.941967 20989 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:13.948693 20989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce: Generated new TSK 0
I20260926 13:44:13.949602 20989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:13.953642 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.1:0
--local_ip_for_outbound_sockets=127.1.49.1
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:37539
--builtin_ntp_servers=127.1.49.20:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:44:14.075392 20999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:14.075700 20999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:14.075743 20999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:14.080087 20999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:14.080155 20999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:14.080214 20999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.1
I20260926 13:44:14.085656 20999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.49.1
--webserver_port=0
--tserver_master_addrs=127.1.49.62:37539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.1
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:14.086824 20999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:14.088308 20999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:14.098482 21007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:14.098511 21004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:14.098493 21005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:14.098709 20999 server_base.cc:1034] running on GCE node
I20260926 13:44:14.099293 20999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:14.100067 20999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:14.101245 20999 hybrid_clock.cc:648] HybridClock initialized: now 1790430254101213 us; error 48 us; skew 500 ppm
I20260926 13:44:14.103225 20999 webserver.cc:460] Webserver started at http://127.1.49.1:39631/ using document root <none> and password file <none>
I20260926 13:44:14.103904 20999 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:14.103995 20999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:14.104235 20999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:14.106352 20999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/data/instance:
uuid: "3e2b04d3845d453d9a660f5f8b539e94"
format_stamp: "Formatted at 2026-09-26 13:44:14 on dist-test-slave-1wns"
server_key: "030b15b838edd54130aa50f279cc5439"
server_key_iv: "b6ec2516011e3622503ee0f2d77f6d62"
server_key_version: "encryptionkey@0"
I20260926 13:44:14.106936 20999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance:
uuid: "3e2b04d3845d453d9a660f5f8b539e94"
format_stamp: "Formatted at 2026-09-26 13:44:14 on dist-test-slave-1wns"
server_key: "030b15b838edd54130aa50f279cc5439"
server_key_iv: "b6ec2516011e3622503ee0f2d77f6d62"
server_key_version: "encryptionkey@0"
I20260926 13:44:14.111025 20999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 13:44:14.113878 21013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:14.115500 20999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:44:14.115664 20999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/wal
uuid: "3e2b04d3845d453d9a660f5f8b539e94"
format_stamp: "Formatted at 2026-09-26 13:44:14 on dist-test-slave-1wns"
server_key: "030b15b838edd54130aa50f279cc5439"
server_key_iv: "b6ec2516011e3622503ee0f2d77f6d62"
server_key_version: "encryptionkey@0"
I20260926 13:44:14.115851 20999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:14.134856 20999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:14.135648 20999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:14.135938 20999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:14.136574 20999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:14.137598 20999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:14.137668 20999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:14.137744 20999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:14.137786 20999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:14.147405 20999 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.1:34223
I20260926 13:44:14.147425 21126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.1:34223 every 8 connection(s)
I20260926 13:44:14.148474 20999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/data/info.pb
I20260926 13:44:14.150790 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 20999
I20260926 13:44:14.150933 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-0/wal/instance
I20260926 13:44:14.151247 1220 external_mini_cluster.cc:1428] Setting key 29213f9212c7ff6b1a807ad853e67e13
I20260926 13:44:14.153465 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.2:0
--local_ip_for_outbound_sockets=127.1.49.2
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:37539
--builtin_ntp_servers=127.1.49.20:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:44:14.158674 20978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.150413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.1:45951 (local address 127.1.49.62:37539)
0926 13:44:14.150596 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:14.150603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:14.152048 (+ 1445us) server_negotiation.cc:407] Connection header received
0926 13:44:14.153502 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:14.153505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:14.153567 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:14.153652 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:14.154614 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.155156 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.156444 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.156676 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.156795 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.158156 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:14.158159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:14.158161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:14.158175 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:14.158196 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:14.158204 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:14.158283 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:14.158445 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 13:44:14.158520 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 13:44:14.159612 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.150786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:37539 (local address 127.1.49.1:45951)
0926 13:44:14.151623 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:14.151678 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:14.153271 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:14.153840 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:14.153848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:14.153953 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:14.154467 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.154480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.155290 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.155294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.156307 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.156318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.156844 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.156848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.157116 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.157835 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:44:14.157837 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:14.157882 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:44:14.157987 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:14.158285 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:14.158297 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:14.158304 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:44:14.158444 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 13:44:14.158763 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":178,"threads_started":1}
I20260926 13:44:14.161057 21127 heartbeater.cc:347] Connected to a master server at 127.1.49.62:37539
I20260926 13:44:14.161320 21127 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:14.161878 21127 heartbeater.cc:511] Master 127.1.49.62:37539 requested a full tablet report, sending...
I20260926 13:44:14.163719 20939 ts_manager.cc:194] Registered new tserver with Master: 3e2b04d3845d453d9a660f5f8b539e94 (127.1.49.1:34223)
I20260926 13:44:14.165450 20939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.49.1:45951
W20260926 13:44:14.274852 21130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:14.275170 21130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:14.275211 21130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:14.279485 21130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:14.279563 21130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:14.279623 21130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.2
I20260926 13:44:14.284731 21130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.49.2
--webserver_port=0
--tserver_master_addrs=127.1.49.62:37539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.2
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:14.285880 21130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:14.287325 21130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:14.298302 21135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:14.298346 21138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:14.298470 21130 server_base.cc:1034] running on GCE node
W20260926 13:44:14.298301 21136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:14.299095 21130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:14.299857 21130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:14.301084 21130 hybrid_clock.cc:648] HybridClock initialized: now 1790430254301069 us; error 35 us; skew 500 ppm
I20260926 13:44:14.303247 21130 webserver.cc:460] Webserver started at http://127.1.49.2:34871/ using document root <none> and password file <none>
I20260926 13:44:14.303949 21130 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:14.304044 21130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:14.304287 21130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:14.306528 21130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/data/instance:
uuid: "6d310d81469e4f30afc4fc7f3eebfe8a"
format_stamp: "Formatted at 2026-09-26 13:44:14 on dist-test-slave-1wns"
server_key: "71668e27dcf90672bb469a049186a3b3"
server_key_iv: "1a377a27a2d196bd91b23e032b19270d"
server_key_version: "encryptionkey@0"
I20260926 13:44:14.307137 21130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance:
uuid: "6d310d81469e4f30afc4fc7f3eebfe8a"
format_stamp: "Formatted at 2026-09-26 13:44:14 on dist-test-slave-1wns"
server_key: "71668e27dcf90672bb469a049186a3b3"
server_key_iv: "1a377a27a2d196bd91b23e032b19270d"
server_key_version: "encryptionkey@0"
I20260926 13:44:14.311674 21130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 13:44:14.315121 21144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:14.317132 21130 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 13:44:14.317327 21130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/wal
uuid: "6d310d81469e4f30afc4fc7f3eebfe8a"
format_stamp: "Formatted at 2026-09-26 13:44:14 on dist-test-slave-1wns"
server_key: "71668e27dcf90672bb469a049186a3b3"
server_key_iv: "1a377a27a2d196bd91b23e032b19270d"
server_key_version: "encryptionkey@0"
I20260926 13:44:14.317482 21130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:14.346602 21130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:14.347427 21130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:14.347640 21130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:14.348363 21130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:14.349617 21130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:14.349709 21130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:14.349804 21130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:14.349852 21130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:14.360416 21130 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.2:36843
I20260926 13:44:14.360446 21257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.2:36843 every 8 connection(s)
I20260926 13:44:14.361498 21130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/data/info.pb
I20260926 13:44:14.371474 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 21130
I20260926 13:44:14.371650 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-1/wal/instance
I20260926 13:44:14.372164 1220 external_mini_cluster.cc:1428] Setting key 5b4ca40df6d32c58916cb02ebbac8999
I20260926 13:44:14.372881 20978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.363681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.2:43679 (local address 127.1.49.62:37539)
0926 13:44:14.363939 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:14.363943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:14.365332 (+ 1389us) server_negotiation.cc:407] Connection header received
0926 13:44:14.366712 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:14.366715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:14.366770 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:14.366879 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:14.368145 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.368802 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.370316 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.370573 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.370667 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.372278 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:14.372282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:14.372285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:14.372300 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:14.372325 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:14.372339 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:14.372400 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:14.372608 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 13:44:14.372714 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:44:14.374083 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.364210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:37539 (local address 127.1.49.2:43679)
0926 13:44:14.365047 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:14.365093 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:14.366465 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:14.367108 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:14.367119 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:14.367235 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:14.367951 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.367976 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.368990 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.368994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.370138 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.370150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.370794 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.370798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.370973 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.371891 (+ 918us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:44:14.371894 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:14.371967 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:44:14.372082 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:14.372429 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:14.372440 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:14.372447 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:44:14.372593 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 13:44:14.372953 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":128,"threads_started":1}
I20260926 13:44:14.374567 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.49.3:0
--local_ip_for_outbound_sockets=127.1.49.3
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:37539
--builtin_ntp_servers=127.1.49.20:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:44:14.375275 21258 heartbeater.cc:347] Connected to a master server at 127.1.49.62:37539
I20260926 13:44:14.375553 21258 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:14.376319 21258 heartbeater.cc:511] Master 127.1.49.62:37539 requested a full tablet report, sending...
I20260926 13:44:14.377530 20939 ts_manager.cc:194] Registered new tserver with Master: 6d310d81469e4f30afc4fc7f3eebfe8a (127.1.49.2:36843)
I20260926 13:44:14.378207 20939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.49.2:43679
W20260926 13:44:14.497960 21261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:14.498312 21261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:14.498402 21261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:14.503368 21261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:14.503479 21261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:14.503582 21261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.49.3
I20260926 13:44:14.509168 21261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.49.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.49.3
--webserver_port=0
--tserver_master_addrs=127.1.49.62:37539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.49.3
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:14.510421 21261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:14.512075 21261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:14.523532 21267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:14.523568 21266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:14.523535 21269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:14.523921 21261 server_base.cc:1034] running on GCE node
I20260926 13:44:14.524534 21261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:14.525288 21261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:14.526501 21261 hybrid_clock.cc:648] HybridClock initialized: now 1790430254526473 us; error 42 us; skew 500 ppm
I20260926 13:44:14.528790 21261 webserver.cc:460] Webserver started at http://127.1.49.3:40489/ using document root <none> and password file <none>
I20260926 13:44:14.529505 21261 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:14.529614 21261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:14.529877 21261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:14.532399 21261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/data/instance:
uuid: "f5d40b03ab2144e293ea19dabef139e9"
format_stamp: "Formatted at 2026-09-26 13:44:14 on dist-test-slave-1wns"
server_key: "f8525a30d87e6d187e1f8879288ef474"
server_key_iv: "cfc2158910edca6ac7ecea9ba409990a"
server_key_version: "encryptionkey@0"
I20260926 13:44:14.533070 21261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance:
uuid: "f5d40b03ab2144e293ea19dabef139e9"
format_stamp: "Formatted at 2026-09-26 13:44:14 on dist-test-slave-1wns"
server_key: "f8525a30d87e6d187e1f8879288ef474"
server_key_iv: "cfc2158910edca6ac7ecea9ba409990a"
server_key_version: "encryptionkey@0"
I20260926 13:44:14.537595 21261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:44:14.540582 21275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:14.542106 21261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:14.542289 21261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/wal
uuid: "f5d40b03ab2144e293ea19dabef139e9"
format_stamp: "Formatted at 2026-09-26 13:44:14 on dist-test-slave-1wns"
server_key: "f8525a30d87e6d187e1f8879288ef474"
server_key_iv: "cfc2158910edca6ac7ecea9ba409990a"
server_key_version: "encryptionkey@0"
I20260926 13:44:14.542426 21261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:14.557394 21261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:14.558176 21261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:14.558387 21261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:14.559039 21261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:14.560074 21261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:14.560148 21261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:14.560218 21261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:14.560268 21261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:14.570092 21388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.3:39599 every 8 connection(s)
I20260926 13:44:14.570092 21261 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.3:39599
I20260926 13:44:14.571256 21261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/data/info.pb
I20260926 13:44:14.572372 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 21261
I20260926 13:44:14.572464 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/wal/instance
I20260926 13:44:14.572736 1220 external_mini_cluster.cc:1428] Setting key d278701af25447325435a25302a4de5e
I20260926 13:44:14.581526 20978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.573362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:56195 (local address 127.1.49.62:37539)
0926 13:44:14.573522 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:14.573525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:14.574978 (+ 1453us) server_negotiation.cc:407] Connection header received
0926 13:44:14.576468 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:14.576472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:14.576526 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:14.576605 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:14.577542 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.578082 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.579286 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.579531 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.579645 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.580988 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:14.580991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:14.580994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:14.581010 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:14.581039 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:14.581047 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:14.581120 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:14.581303 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 13:44:14.581370 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 13:44:14.582423 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.573950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:37539 (local address 127.1.49.3:56195)
0926 13:44:14.574823 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:14.574878 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:14.576251 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:14.576786 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:14.576793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:14.576885 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:14.577396 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.577410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.578211 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.578220 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.579150 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.579161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.579690 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.579694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.579946 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.580614 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:44:14.580615 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:14.580659 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:44:14.580739 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:14.581134 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:14.581145 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:14.581155 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 13:44:14.581280 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 13:44:14.581597 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":177,"threads_started":1}
I20260926 13:44:14.583920 21389 heartbeater.cc:347] Connected to a master server at 127.1.49.62:37539
I20260926 13:44:14.584272 21389 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:14.584880 21389 heartbeater.cc:511] Master 127.1.49.62:37539 requested a full tablet report, sending...
I20260926 13:44:14.586084 20938 ts_manager.cc:194] Registered new tserver with Master: f5d40b03ab2144e293ea19dabef139e9 (127.1.49.3:39599)
I20260926 13:44:14.586908 1220 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:44:14.586886 20938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.49.3:56195
I20260926 13:44:14.593068 20978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.589091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.1.49.62:37539)
0926 13:44:14.589214 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:14.589219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:14.589457 (+ 238us) server_negotiation.cc:407] Connection header received
0926 13:44:14.589626 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:14.589634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:14.589677 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:14.589758 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:14.590050 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.590565 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.591433 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.591648 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.591761 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.592653 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:14.592657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:14.592659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:14.592671 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:14.592688 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:14.592696 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:14.592770 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:14.592861 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 13:44:14.592920 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:44:14.596192 20938 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:44:14.599249 20938 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:44:14.615307 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.608713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.3:39599 (local address 127.0.0.1:36044)
0926 13:44:14.609175 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:14.609236 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:14.609371 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:14.610343 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:14.610349 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:14.610373 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:14.610531 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.610540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.612044 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.612051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.613516 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.613529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.614631 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.614634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.614778 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.614816 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 13:44:14.614959 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 13:44:14.615112 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":35,"thread_start_us":173,"threads_started":1}
I20260926 13:44:14.615312 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.608841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.1.49.3:39599)
0926 13:44:14.609864 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:14.609878 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:14.609904 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:44:14.610002 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:14.610006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:14.610173 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:14.610342 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:14.610677 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.611918 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.613690 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.614441 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.614609 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.614848 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:14.615006 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 13:44:14.615129 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":170,"threads_started":1}
I20260926 13:44:14.616958 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.608734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:34223 (local address 127.0.0.1:42220)
0926 13:44:14.609296 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:14.609309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:14.609371 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:14.611003 (+ 1632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:14.611006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:14.611022 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:14.611153 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.611163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.612693 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.612698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.614151 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.614163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.616513 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.616518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.616615 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.616638 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:44:14.616708 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 13:44:14.616791 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":153,"thread_start_us":131,"threads_started":1}
I20260926 13:44:14.616822 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.609808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.1.49.2:36843)
0926 13:44:14.610265 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:14.610282 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:14.610732 (+ 450us) server_negotiation.cc:407] Connection header received
0926 13:44:14.610869 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:14.610873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:14.611059 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:14.611217 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:14.611474 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.612625 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.614305 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.615259 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.615581 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.615797 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:14.616479 (+ 682us) server_negotiation.cc:299] Negotiation successful
0926 13:44:14.616609 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":130,"threads_started":1}
I20260926 13:44:14.617472 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.609593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:36843 (local address 127.0.0.1:49634)
0926 13:44:14.610618 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:14.610632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:14.610743 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:14.611224 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:14.611227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:14.611246 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:14.611348 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.611355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.612740 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.612743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.614151 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.614163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.615905 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.615909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.616052 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.616079 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 13:44:14.616164 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:44:14.617326 (+ 1162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":134,"thread_start_us":91,"threads_started":1}
I20260926 13:44:14.617872 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.609660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42220 (local address 127.1.49.1:34223)
0926 13:44:14.610497 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:14.610511 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:14.610538 (+ 27us) server_negotiation.cc:407] Connection header received
0926 13:44:14.610625 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:14.610633 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:14.610845 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:14.611032 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:14.611294 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.612518 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.614940 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.616166 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.616272 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.617513 (+ 1241us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:14.617583 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 13:44:14.617711 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":142,"threads_started":1}
I20260926 13:44:14.619256 21323 tablet_service.cc:1467] Processing CreateTablet for tablet b82572258be1468fbd5db298c0e89d14 (DEFAULT_TABLE table=test-table [id=fe8f0d4fe8f846fb91367e4c56d9515c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:14.620604 21323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82572258be1468fbd5db298c0e89d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:14.620625 21061 tablet_service.cc:1467] Processing CreateTablet for tablet b82572258be1468fbd5db298c0e89d14 (DEFAULT_TABLE table=test-table [id=fe8f0d4fe8f846fb91367e4c56d9515c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:14.620733 21192 tablet_service.cc:1467] Processing CreateTablet for tablet b82572258be1468fbd5db298c0e89d14 (DEFAULT_TABLE table=test-table [id=fe8f0d4fe8f846fb91367e4c56d9515c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:14.621789 21061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82572258be1468fbd5db298c0e89d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:14.621789 21192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82572258be1468fbd5db298c0e89d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:14.628548 21406 tablet_bootstrap.cc:492] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94: Bootstrap starting.
I20260926 13:44:14.629930 21408 tablet_bootstrap.cc:492] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9: Bootstrap starting.
I20260926 13:44:14.630666 21407 tablet_bootstrap.cc:492] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a: Bootstrap starting.
I20260926 13:44:14.632055 21406 tablet_bootstrap.cc:654] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:14.632089 21408 tablet_bootstrap.cc:654] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:14.632911 21408 log.cc:824] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:14.632988 21406 log.cc:824] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:14.633160 21407 tablet_bootstrap.cc:654] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:14.634047 21407 log.cc:824] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:14.635300 21408 tablet_bootstrap.cc:492] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9: No bootstrap required, opened a new log
I20260926 13:44:14.635324 21406 tablet_bootstrap.cc:492] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94: No bootstrap required, opened a new log
I20260926 13:44:14.635535 21406 ts_tablet_manager.cc:1397] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260926 13:44:14.635535 21408 ts_tablet_manager.cc:1397] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:44:14.636461 21407 tablet_bootstrap.cc:492] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a: No bootstrap required, opened a new log
I20260926 13:44:14.636677 21407 ts_tablet_manager.cc:1397] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:44:14.640341 21407 raft_consensus.cc:348] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d40b03ab2144e293ea19dabef139e9" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39599 } } peers { permanent_uuid: "3e2b04d3845d453d9a660f5f8b539e94" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34223 } } peers { permanent_uuid: "6d310d81469e4f30afc4fc7f3eebfe8a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36843 } }
I20260926 13:44:14.640631 21407 raft_consensus.cc:374] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:14.640690 21407 raft_consensus.cc:729] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d310d81469e4f30afc4fc7f3eebfe8a, State: Initialized, Role: FOLLOWER
I20260926 13:44:14.640515 21408 raft_consensus.cc:348] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d40b03ab2144e293ea19dabef139e9" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39599 } } peers { permanent_uuid: "3e2b04d3845d453d9a660f5f8b539e94" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34223 } } peers { permanent_uuid: "6d310d81469e4f30afc4fc7f3eebfe8a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36843 } }
I20260926 13:44:14.640515 21406 raft_consensus.cc:348] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d40b03ab2144e293ea19dabef139e9" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39599 } } peers { permanent_uuid: "3e2b04d3845d453d9a660f5f8b539e94" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34223 } } peers { permanent_uuid: "6d310d81469e4f30afc4fc7f3eebfe8a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36843 } }
I20260926 13:44:14.640786 21408 raft_consensus.cc:374] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:14.640786 21406 raft_consensus.cc:374] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:14.640861 21408 raft_consensus.cc:729] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d40b03ab2144e293ea19dabef139e9, State: Initialized, Role: FOLLOWER
I20260926 13:44:14.640861 21406 raft_consensus.cc:729] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2b04d3845d453d9a660f5f8b539e94, State: Initialized, Role: FOLLOWER
I20260926 13:44:14.641289 21407 consensus_queue.cc:260] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d40b03ab2144e293ea19dabef139e9" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39599 } } peers { permanent_uuid: "3e2b04d3845d453d9a660f5f8b539e94" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34223 } } peers { permanent_uuid: "6d310d81469e4f30afc4fc7f3eebfe8a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36843 } }
I20260926 13:44:14.641357 21406 consensus_queue.cc:260] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d40b03ab2144e293ea19dabef139e9" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39599 } } peers { permanent_uuid: "3e2b04d3845d453d9a660f5f8b539e94" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34223 } } peers { permanent_uuid: "6d310d81469e4f30afc4fc7f3eebfe8a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36843 } }
I20260926 13:44:14.641357 21408 consensus_queue.cc:260] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d40b03ab2144e293ea19dabef139e9" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39599 } } peers { permanent_uuid: "3e2b04d3845d453d9a660f5f8b539e94" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34223 } } peers { permanent_uuid: "6d310d81469e4f30afc4fc7f3eebfe8a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36843 } }
I20260926 13:44:14.642066 21258 heartbeater.cc:503] Master 127.1.49.62:37539 was elected leader, sending a full tablet report...
I20260926 13:44:14.642486 21407 ts_tablet_manager.cc:1428] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:44:14.642491 21406 ts_tablet_manager.cc:1428] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94: Time spent starting tablet: real 0.007s user 0.000s sys 0.008s
I20260926 13:44:14.642541 21408 ts_tablet_manager.cc:1428] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260926 13:44:14.642859 21389 heartbeater.cc:503] Master 127.1.49.62:37539 was elected leader, sending a full tablet report...
I20260926 13:44:14.642859 21127 heartbeater.cc:503] Master 127.1.49.62:37539 was elected leader, sending a full tablet report...
W20260926 13:44:14.649673 21128 tablet.cc:2369] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:44:14.822300 21390 tablet.cc:2369] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:44:14.862855 21259 tablet.cc:2369] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:44:14.899910 21413 raft_consensus.cc:482] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:44:14.900171 21413 raft_consensus.cc:504] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d40b03ab2144e293ea19dabef139e9" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39599 } } peers { permanent_uuid: "3e2b04d3845d453d9a660f5f8b539e94" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34223 } } peers { permanent_uuid: "6d310d81469e4f30afc4fc7f3eebfe8a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36843 } }
I20260926 13:44:14.901624 21413 leader_election.cc:288] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e2b04d3845d453d9a660f5f8b539e94 (127.1.49.1:34223), 6d310d81469e4f30afc4fc7f3eebfe8a (127.1.49.2:36843)
I20260926 13:44:14.906386 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.902017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:45049 (local address 127.1.49.1:34223)
0926 13:44:14.902142 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:14.902147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:14.902653 (+ 506us) server_negotiation.cc:407] Connection header received
0926 13:44:14.902765 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:14.902768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:14.902823 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:14.902984 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:14.903146 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.903886 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.905059 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.905598 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.905708 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.905780 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:14.906120 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 13:44:14.906239 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:44:14.906523 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.901881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:36843 (local address 127.1.49.3:48799)
0926 13:44:14.902126 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:14.902143 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:14.902249 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:14.902534 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:14.902537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:14.902562 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:14.902698 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.902706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.903822 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.903827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.905182 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.905197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.906067 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.906071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.906187 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.906211 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 13:44:14.906282 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 13:44:14.906361 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260926 13:44:14.906955 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.902094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.49.3:48799 (local address 127.1.49.2:36843)
0926 13:44:14.902212 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:14.902216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:14.902231 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:44:14.902373 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:14.902377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:14.902435 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:14.902516 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:14.902848 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.903669 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.905332 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.905944 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.906062 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.906116 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:14.906703 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 13:44:14.906796 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:44:14.906961 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.901881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:34223 (local address 127.1.49.3:45049)
0926 13:44:14.902563 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:14.902578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:14.902677 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:14.902919 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:14.902922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:14.902939 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:14.903056 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.903062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.904005 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.904009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.904922 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:14.904933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.905794 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.905799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:14.905922 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.905965 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 13:44:14.906637 (+ 672us) client_negotiation.cc:240] Negotiation successful
0926 13:44:14.906728 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":289,"thread_start_us":137,"threads_started":1}
I20260926 13:44:14.907559 21212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82572258be1468fbd5db298c0e89d14" candidate_uuid: "f5d40b03ab2144e293ea19dabef139e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d310d81469e4f30afc4fc7f3eebfe8a" is_pre_election: true
I20260926 13:44:14.907588 21081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82572258be1468fbd5db298c0e89d14" candidate_uuid: "f5d40b03ab2144e293ea19dabef139e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2b04d3845d453d9a660f5f8b539e94" is_pre_election: true
I20260926 13:44:14.907940 21212 raft_consensus.cc:2454] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d40b03ab2144e293ea19dabef139e9 in term 0.
I20260926 13:44:14.907940 21081 raft_consensus.cc:2454] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d40b03ab2144e293ea19dabef139e9 in term 0.
I20260926 13:44:14.908509 21279 leader_election.cc:302] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3e2b04d3845d453d9a660f5f8b539e94, 6d310d81469e4f30afc4fc7f3eebfe8a, f5d40b03ab2144e293ea19dabef139e9; no voters:
I20260926 13:44:14.908840 21413 raft_consensus.cc:2781] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:44:14.908936 21413 raft_consensus.cc:482] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:44:14.908991 21413 raft_consensus.cc:3037] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:14.910125 21413 raft_consensus.cc:504] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d40b03ab2144e293ea19dabef139e9" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39599 } } peers { permanent_uuid: "3e2b04d3845d453d9a660f5f8b539e94" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34223 } } peers { permanent_uuid: "6d310d81469e4f30afc4fc7f3eebfe8a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36843 } }
I20260926 13:44:14.910538 21413 leader_election.cc:288] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [CANDIDATE]: Term 1 election: Requested vote from peers 3e2b04d3845d453d9a660f5f8b539e94 (127.1.49.1:34223), 6d310d81469e4f30afc4fc7f3eebfe8a (127.1.49.2:36843)
I20260926 13:44:14.910975 21081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82572258be1468fbd5db298c0e89d14" candidate_uuid: "f5d40b03ab2144e293ea19dabef139e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2b04d3845d453d9a660f5f8b539e94"
I20260926 13:44:14.910969 21212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82572258be1468fbd5db298c0e89d14" candidate_uuid: "f5d40b03ab2144e293ea19dabef139e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d310d81469e4f30afc4fc7f3eebfe8a"
I20260926 13:44:14.911131 21212 raft_consensus.cc:3037] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:14.911130 21081 raft_consensus.cc:3037] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:14.912549 21081 raft_consensus.cc:2454] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d40b03ab2144e293ea19dabef139e9 in term 1.
I20260926 13:44:14.912549 21212 raft_consensus.cc:2454] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d40b03ab2144e293ea19dabef139e9 in term 1.
I20260926 13:44:14.912990 21277 leader_election.cc:302] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d310d81469e4f30afc4fc7f3eebfe8a, f5d40b03ab2144e293ea19dabef139e9; no voters:
I20260926 13:44:14.913185 21413 raft_consensus.cc:2781] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:14.913455 21413 raft_consensus.cc:686] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [term 1 LEADER]: Becoming Leader. State: Replica: f5d40b03ab2144e293ea19dabef139e9, State: Running, Role: LEADER
I20260926 13:44:14.913826 21413 consensus_queue.cc:237] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d40b03ab2144e293ea19dabef139e9" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39599 } } peers { permanent_uuid: "3e2b04d3845d453d9a660f5f8b539e94" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34223 } } peers { permanent_uuid: "6d310d81469e4f30afc4fc7f3eebfe8a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36843 } }
I20260926 13:44:14.917609 20939 catalog_manager.cc:5495] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5d40b03ab2144e293ea19dabef139e9 (127.1.49.3). New cstate: current_term: 1 leader_uuid: "f5d40b03ab2144e293ea19dabef139e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d40b03ab2144e293ea19dabef139e9" member_type: VOTER last_known_addr { host: "127.1.49.3" port: 39599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e2b04d3845d453d9a660f5f8b539e94" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d310d81469e4f30afc4fc7f3eebfe8a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36843 } health_report { overall_health: UNKNOWN } } }
I20260926 13:44:14.987128 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:14.983137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.1.49.3:39599)
0926 13:44:14.983304 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:14.983310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:14.983327 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:44:14.983465 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:14.983468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:14.983536 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:14.983660 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:14.983946 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.984516 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.985475 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:14.985729 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:14.985858 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:14.986197 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:14.986663 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:14.986820 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:14.986881 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 13:44:14.986940 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 13:44:14.993685 21212 raft_consensus.cc:1261] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a [term 1 FOLLOWER]: Refusing update from remote peer f5d40b03ab2144e293ea19dabef139e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:14.993710 21081 raft_consensus.cc:1261] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94 [term 1 FOLLOWER]: Refusing update from remote peer f5d40b03ab2144e293ea19dabef139e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:14.994570 21416 consensus_queue.cc:1035] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d310d81469e4f30afc4fc7f3eebfe8a" member_type: VOTER last_known_addr { host: "127.1.49.2" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:14.994796 21413 consensus_queue.cc:1035] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2b04d3845d453d9a660f5f8b539e94" member_type: VOTER last_known_addr { host: "127.1.49.1" port: 34223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:15.002367 21419 mvcc.cc:205] Tried to move back new op lower bound from 7333602324445896704 to 7333602324129906688. Current Snapshot: MvccSnapshot[applied={T|T < 7333602324445896704}]
I20260926 13:44:15.006547 21421 mvcc.cc:205] Tried to move back new op lower bound from 7333602324445896704 to 7333602324129906688. Current Snapshot: MvccSnapshot[applied={T|T < 7333602324445896704 or (T in {7333602324445896704})}]
I20260926 13:44:15.011988 20939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:15.012173 20939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:15.015101 20939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e9adacf16d54c788cb0f9127eb14bce: Sending DeleteTablet for 3 replicas of tablet b82572258be1468fbd5db298c0e89d14
I20260926 13:44:15.016141 21061 tablet_service.cc:1514] Processing DeleteTablet for tablet b82572258be1468fbd5db298c0e89d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:15 UTC) from {username='slave'} at 127.0.0.1:42220
I20260926 13:44:15.016223 21323 tablet_service.cc:1514] Processing DeleteTablet for tablet b82572258be1468fbd5db298c0e89d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:15 UTC) from {username='slave'} at 127.0.0.1:36044
I20260926 13:44:15.016901 21428 tablet_replica.cc:330] T b82572258be1468fbd5db298c0e89d14 P 3e2b04d3845d453d9a660f5f8b539e94: stopping tablet replica
I20260926 13:44:15.016928 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 20999
I20260926 13:44:15.017603 21192 tablet_service.cc:1514] Processing DeleteTablet for tablet b82572258be1468fbd5db298c0e89d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:15 UTC) from {username='slave'} at 127.0.0.1:49634
I20260926 13:44:15.018834 21430 tablet_replica.cc:330] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a: stopping tablet replica
I20260926 13:44:15.019260 21430 raft_consensus.cc:2229] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:44:15.019830 21430 raft_consensus.cc:2258] T b82572258be1468fbd5db298c0e89d14 P 6d310d81469e4f30afc4fc7f3eebfe8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:15.020311 21429 tablet_replica.cc:330] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9: stopping tablet replica
I20260926 13:44:15.020690 21429 raft_consensus.cc:2229] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:44:15.021191 21429 raft_consensus.cc:2258] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:15.022956 21429 ts_tablet_manager.cc:1905] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 13:44:15.026278 20926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.49.1:34223 (error 108)
I20260926 13:44:15.026484 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 21130
I20260926 13:44:15.026047 21429 ts_tablet_manager.cc:1918] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:44:15.028839 21429 log.cc:1196] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9: Deleting WAL directory at /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790429983839632-1220-0/minicluster-data/ts-2/wal/wals/b82572258be1468fbd5db298c0e89d14
I20260926 13:44:15.029081 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:15.028660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.1:34223 (local address 127.0.0.1:42232)
0926 13:44:15.028834 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:15.028950 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.1:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 13:44:15.029348 20926 catalog_manager.cc:4584] TS 3e2b04d3845d453d9a660f5f8b539e94 (127.1.49.1:34223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b82572258be1468fbd5db298c0e89d14: Network error: Client connection negotiation failed: client connection to 127.1.49.1:34223: connect: Connection refused (error 111)
I20260926 13:44:15.029796 21429 ts_tablet_manager.cc:1939] T b82572258be1468fbd5db298c0e89d14 P f5d40b03ab2144e293ea19dabef139e9: Deleting consensus metadata
I20260926 13:44:15.031172 20924 catalog_manager.cc:4841] TS f5d40b03ab2144e293ea19dabef139e9 (127.1.49.3:39599): tablet b82572258be1468fbd5db298c0e89d14 (table test-table [id=fe8f0d4fe8f846fb91367e4c56d9515c]) successfully deleted
I20260926 13:44:15.033691 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 21261
I20260926 13:44:15.035444 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:15.034535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.2:36843 (local address 127.0.0.1:49638)
0926 13:44:15.035310 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:15.035363 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.2:36843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":695}
I20260926 13:44:15.040381 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 20908
2026-09-26T13:44:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1380 ms)
[----------] 6 tests from EncryptionPolicyTest (11204 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:15 dist-test-slave-1wns krb5kdc[21434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:15 dist-test-slave-1wns krb5kdc[21434](info): set up 2 sockets
Sep 26 13:44:15 dist-test-slave-1wns krb5kdc[21434](info): commencing operation
krb5kdc: starting...
W20260926 13:44:16.132226 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:16 dist-test-slave-1wns krb5kdc[21434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:44:16.263932 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:16.380394 21447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:16.380690 21447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:16.380734 21447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:16.384963 21447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:16.385046 21447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:16.385066 21447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:16.385084 21447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:16.385100 21447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:16.390722 21447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39455
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:16.392024 21447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:16.393127 21447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:16.402422 21455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:16.402401 21453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:16.402400 21452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:16.402704 21447 server_base.cc:1034] running on GCE node
I20260926 13:44:16.403537 21447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:16.405004 21447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:16.406230 21447 hybrid_clock.cc:648] HybridClock initialized: now 1790430256406200 us; error 53 us; skew 500 ppm
Sep 26 13:44:16 dist-test-slave-1wns krb5kdc[21434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:16.409389 21447 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:44:16.410440 21447 webserver.cc:460] Webserver started at http://127.0.0.1:46645/ using document root <none> and password file <none>
I20260926 13:44:16.411041 21447 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:16.411134 21447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:16.411374 21447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:16.413587 21447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "cfd709f3b18e42299179094579dc6d7b"
format_stamp: "Formatted at 2026-09-26 13:44:16 on dist-test-slave-1wns"
server_key: "cfda499fcad1e63977a8d00603787707"
server_key_iv: "e6d182c7c2d9c59afde3da1a7d677744"
server_key_version: "encryptionkey@0"
I20260926 13:44:16.414197 21447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "cfd709f3b18e42299179094579dc6d7b"
format_stamp: "Formatted at 2026-09-26 13:44:16 on dist-test-slave-1wns"
server_key: "cfda499fcad1e63977a8d00603787707"
server_key_iv: "e6d182c7c2d9c59afde3da1a7d677744"
server_key_version: "encryptionkey@0"
I20260926 13:44:16.418385 21447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 13:44:16.421209 21462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:16.422598 21447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:44:16.422765 21447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "cfd709f3b18e42299179094579dc6d7b"
format_stamp: "Formatted at 2026-09-26 13:44:16 on dist-test-slave-1wns"
server_key: "cfda499fcad1e63977a8d00603787707"
server_key_iv: "e6d182c7c2d9c59afde3da1a7d677744"
server_key_version: "encryptionkey@0"
I20260926 13:44:16.422976 21447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:16.447091 21447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:16.450614 21447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:16.450917 21447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:16.458917 21447 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39455
I20260926 13:44:16.458953 21514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39455 every 8 connection(s)
I20260926 13:44:16.460112 21447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:16.461079 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 21447
I20260926 13:44:16.461216 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:16.461505 1220 external_mini_cluster.cc:1428] Setting key e5f063b5e0fbcc135d82fa2c29525d2d
I20260926 13:44:16.463733 21515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:16.470489 21515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b: Bootstrap starting.
Sep 26 13:44:16 dist-test-slave-1wns krb5kdc[21434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:44:16.473502 21515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:16.474460 21515 log.cc:824] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:16.477108 21515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b: No bootstrap required, opened a new log
I20260926 13:44:16.478535 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:16.463069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34569 (local address 127.0.0.1:39455)
0926 13:44:16.463677 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:16.463695 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:16.463741 (+ 46us) server_negotiation.cc:407] Connection header received
0926 13:44:16.464553 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:16.464573 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:16.464990 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:16.465431 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:16.466357 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:16.467575 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:16.468675 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:16.469231 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:16.469457 (+ 226us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:16.472802 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:16.472809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:16.472825 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:16.472864 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:16.475044 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:16.475846 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:16.475852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:16.475857 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:16.476037 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:16.476405 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:16.476407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:16.476408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:16.476835 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:16.477060 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:16.477391 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 13:44:16.477659 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":170,"threads_started":1}
I20260926 13:44:16.481134 21515 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd709f3b18e42299179094579dc6d7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39455 } }
I20260926 13:44:16.481398 21515 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:16.481484 21515 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd709f3b18e42299179094579dc6d7b, State: Initialized, Role: FOLLOWER
I20260926 13:44:16.481801 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:16.481995 21515 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd709f3b18e42299179094579dc6d7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39455 } }
I20260926 13:44:16.482148 21515 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:16.482215 21515 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:16.482314 21515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:16.483404 21515 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd709f3b18e42299179094579dc6d7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39455 } }
I20260926 13:44:16.483831 21515 leader_election.cc:302] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfd709f3b18e42299179094579dc6d7b; no voters:
I20260926 13:44:16.484192 21515 leader_election.cc:288] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:16.484300 21525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:16.484550 21525 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [term 1 LEADER]: Becoming Leader. State: Replica: cfd709f3b18e42299179094579dc6d7b, State: Running, Role: LEADER
I20260926 13:44:16.484870 21525 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd709f3b18e42299179094579dc6d7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39455 } }
I20260926 13:44:16.485517 21515 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:16.487010 21526 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfd709f3b18e42299179094579dc6d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd709f3b18e42299179094579dc6d7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39455 } } }
I20260926 13:44:16.487057 21527 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfd709f3b18e42299179094579dc6d7b. Latest consensus state: current_term: 1 leader_uuid: "cfd709f3b18e42299179094579dc6d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd709f3b18e42299179094579dc6d7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39455 } } }
I20260926 13:44:16.487191 21527 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:16.487140 21526 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:16.487633 21538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:16.491235 21538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:16.496704 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:16.484023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50717 (local address 127.0.0.1:39455)
0926 13:44:16.484145 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:16.484164 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:16.486297 (+ 2133us) server_negotiation.cc:407] Connection header received
0926 13:44:16.487402 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:16.487406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:16.487462 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:16.487528 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:16.488577 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:16.489107 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:16.490296 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:16.490642 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:16.490751 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:16.492668 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:16.492672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:16.492675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:16.492706 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:16.494624 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:16.495249 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:16.495253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:16.495255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:16.495350 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:16.495718 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:16.495721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:16.495723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:16.495993 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:16.496150 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:16.496417 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 13:44:16.496525 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:44:16.498059 21538 catalog_manager.cc:1347] Generated new cluster ID: 157584eb37874af6a20dd0066bcf7012
I20260926 13:44:16.498135 21538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:16.506783 21538 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:16.508070 21538 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:16.513073 21538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfd709f3b18e42299179094579dc6d7b: Generated new TSK 0
I20260926 13:44:16.513922 21538 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:16.524245 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 21447
W20260926 13:44:16.529697 21432 connection.cc:535] client connection to 127.0.0.1:39455 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39455): Connection reset by peer (error 104)
2026-09-26T13:44:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1480 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:16 dist-test-slave-1wns krb5kdc[21546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:16 dist-test-slave-1wns krb5kdc[21546](info): set up 2 sockets
Sep 26 13:44:16 dist-test-slave-1wns krb5kdc[21546](info): commencing operation
krb5kdc: starting...
W20260926 13:44:17.624779 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:17 dist-test-slave-1wns krb5kdc[21546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:44:17.759781 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43503
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:17.876070 21559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:17.876408 21559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:17.876514 21559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:17.880769 21559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:17.880877 21559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:17.880913 21559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:17.880964 21559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:17.881016 21559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:17.886629 21559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43503
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:17.887970 21559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:17.889096 21559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:17.898205 21567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:17.898218 21565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:17.898218 21564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:17.898679 21559 server_base.cc:1034] running on GCE node
I20260926 13:44:17.899283 21559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:17.900595 21559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:17.901800 21559 hybrid_clock.cc:648] HybridClock initialized: now 1790430257901792 us; error 49 us; skew 500 ppm
Sep 26 13:44:17 dist-test-slave-1wns krb5kdc[21546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:17.905604 21559 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:44:17.906900 21559 webserver.cc:460] Webserver started at http://127.0.0.1:34437/ using document root <none> and password file <none>
I20260926 13:44:17.907449 21559 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:17.907508 21559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:17.907685 21559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:17.909881 21559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "ad336f990f8f44f09c440eac81610203"
format_stamp: "Formatted at 2026-09-26 13:44:17 on dist-test-slave-1wns"
server_key: "cace1c2a432d31f2c8547ed6720b8bf0"
server_key_iv: "eeebd44fd2437ae339de42dd012aebd1"
server_key_version: "encryptionkey@0"
I20260926 13:44:17.910409 21559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "ad336f990f8f44f09c440eac81610203"
format_stamp: "Formatted at 2026-09-26 13:44:17 on dist-test-slave-1wns"
server_key: "cace1c2a432d31f2c8547ed6720b8bf0"
server_key_iv: "eeebd44fd2437ae339de42dd012aebd1"
server_key_version: "encryptionkey@0"
I20260926 13:44:17.914415 21559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:44:17.917068 21574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:17.918424 21559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:17.918593 21559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "ad336f990f8f44f09c440eac81610203"
format_stamp: "Formatted at 2026-09-26 13:44:17 on dist-test-slave-1wns"
server_key: "cace1c2a432d31f2c8547ed6720b8bf0"
server_key_iv: "eeebd44fd2437ae339de42dd012aebd1"
server_key_version: "encryptionkey@0"
I20260926 13:44:17.918720 21559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:17.938601 21559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:17.942291 21559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:17.942519 21559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:17.949838 21559 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43503
I20260926 13:44:17.949852 21626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43503 every 8 connection(s)
I20260926 13:44:17.950866 21559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:17.953909 21627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:17.957294 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 21559
I20260926 13:44:17.957464 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:17.957769 1220 external_mini_cluster.cc:1428] Setting key e0e4360069071bd8e27e54fc5821a1da
I20260926 13:44:17.960548 21627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203: Bootstrap starting.
I20260926 13:44:17.963534 21627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:17.964489 21627 log.cc:824] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:17.967008 21627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203: No bootstrap required, opened a new log
I20260926 13:44:17.968775 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:17.959407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60113 (local address 127.0.0.1:43503)
0926 13:44:17.959928 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:17.959950 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:17.959996 (+ 46us) server_negotiation.cc:407] Connection header received
0926 13:44:17.960885 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:17.960910 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:17.961376 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:17.961774 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:17.962233 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:17.963757 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:17.964815 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:17.965398 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:17.965578 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:17.966567 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:17.966573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:17.966590 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:17.967126 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:17.967200 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:17.967233 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:17.967399 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:17.967604 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 13:44:17.967747 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":138,"threads_started":1}
I20260926 13:44:17.971608 21627 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad336f990f8f44f09c440eac81610203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43503 } }
I20260926 13:44:17.971958 21627 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:17.972040 21627 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad336f990f8f44f09c440eac81610203, State: Initialized, Role: FOLLOWER
I20260926 13:44:17.972358 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:17.972551 21627 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad336f990f8f44f09c440eac81610203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43503 } }
I20260926 13:44:17.972707 21627 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:17.972783 21627 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:17.972914 21627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:17.974054 21627 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad336f990f8f44f09c440eac81610203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43503 } }
I20260926 13:44:17.974435 21627 leader_election.cc:302] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad336f990f8f44f09c440eac81610203; no voters:
I20260926 13:44:17.974848 21627 leader_election.cc:288] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:17.974977 21636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:17.975220 21636 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [term 1 LEADER]: Becoming Leader. State: Replica: ad336f990f8f44f09c440eac81610203, State: Running, Role: LEADER
I20260926 13:44:17.975569 21636 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad336f990f8f44f09c440eac81610203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43503 } }
I20260926 13:44:17.976384 21627 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:17.977028 21638 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad336f990f8f44f09c440eac81610203. Latest consensus state: current_term: 1 leader_uuid: "ad336f990f8f44f09c440eac81610203" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad336f990f8f44f09c440eac81610203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43503 } } }
I20260926 13:44:17.977185 21638 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:17.977851 21639 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad336f990f8f44f09c440eac81610203" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad336f990f8f44f09c440eac81610203" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43503 } } }
I20260926 13:44:17.977962 21639 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:17.980659 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:17.974831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53495 (local address 127.0.0.1:43503)
0926 13:44:17.974982 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:17.974987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:17.975268 (+ 281us) server_negotiation.cc:407] Connection header received
0926 13:44:17.975479 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:17.975482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:17.975533 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:17.975586 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:17.976099 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:17.976918 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:17.978142 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:17.978508 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:17.978621 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:17.979894 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:17.979898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:17.979901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:17.979916 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:17.979949 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:17.979964 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:17.980055 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:17.980469 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 13:44:17.980508 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 13:44:17.981997 21652 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:17.982074 21652 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:17.982203 21644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:17.982963 21644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:17.989612 21644 catalog_manager.cc:1347] Generated new cluster ID: 305690c62de44bc1a936d671da5c3eec
I20260926 13:44:17.989708 21644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:18.011039 21644 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:18.012245 21644 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:18.021410 21644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad336f990f8f44f09c440eac81610203: Generated new TSK 0
I20260926 13:44:18.022250 21644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:18.030368 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 21559
W20260926 13:44:18.036571 21544 connection.cc:535] client connection to 127.0.0.1:43503 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43503): Connection reset by peer (error 104)
2026-09-26T13:44:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1507 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:18 dist-test-slave-1wns krb5kdc[21658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:18 dist-test-slave-1wns krb5kdc[21658](info): set up 2 sockets
Sep 26 13:44:18 dist-test-slave-1wns krb5kdc[21658](info): commencing operation
krb5kdc: starting...
W20260926 13:44:19.076159 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:19 dist-test-slave-1wns krb5kdc[21658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:44:19.210625 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33039
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:19.333493 21671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:19.333808 21671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:19.333861 21671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:19.338171 21671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:19.338249 21671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:19.338270 21671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:19.338287 21671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:19.338304 21671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:19.343957 21671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33039
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:19.345086 21671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:19.346113 21671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:19.355538 21679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:19.355587 21676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:19.355538 21677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:19.356003 21671 server_base.cc:1034] running on GCE node
I20260926 13:44:19.356652 21671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:19.357963 21671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:19.359191 21671 hybrid_clock.cc:648] HybridClock initialized: now 1790430259359168 us; error 35 us; skew 500 ppm
Sep 26 13:44:19 dist-test-slave-1wns krb5kdc[21658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:19.378288 21671 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:44:19.379431 21671 webserver.cc:460] Webserver started at http://127.0.0.1:34707/ using document root <none> and password file <none>
I20260926 13:44:19.380225 21671 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:19.380330 21671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:19.380604 21671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:19.383064 21671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "d42a0ba788a1449f925a84ad395764df"
format_stamp: "Formatted at 2026-09-26 13:44:19 on dist-test-slave-1wns"
server_key: "bd79ed55812f8287694f5f0e8c8caeb6"
server_key_iv: "f19666ac4a760517c552c1d6ed0b866a"
server_key_version: "encryptionkey@0"
I20260926 13:44:19.383729 21671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "d42a0ba788a1449f925a84ad395764df"
format_stamp: "Formatted at 2026-09-26 13:44:19 on dist-test-slave-1wns"
server_key: "bd79ed55812f8287694f5f0e8c8caeb6"
server_key_iv: "f19666ac4a760517c552c1d6ed0b866a"
server_key_version: "encryptionkey@0"
I20260926 13:44:19.388206 21671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:44:19.391223 21686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:19.392863 21671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:19.393057 21671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "d42a0ba788a1449f925a84ad395764df"
format_stamp: "Formatted at 2026-09-26 13:44:19 on dist-test-slave-1wns"
server_key: "bd79ed55812f8287694f5f0e8c8caeb6"
server_key_iv: "f19666ac4a760517c552c1d6ed0b866a"
server_key_version: "encryptionkey@0"
I20260926 13:44:19.393201 21671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:19.398815 21671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:19.402894 21671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:19.403220 21671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:19.411067 21671 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33039
I20260926 13:44:19.411103 21738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33039 every 8 connection(s)
I20260926 13:44:19.412171 21671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:19.415165 21739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:19.418231 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 21671
I20260926 13:44:19.418429 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:19.418746 1220 external_mini_cluster.cc:1428] Setting key 9753c77fab05a8ad43657524a6a6849c
I20260926 13:44:19.421797 21739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df: Bootstrap starting.
I20260926 13:44:19.424680 21739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:19.425275 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:19.420595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59297 (local address 127.0.0.1:33039)
0926 13:44:19.421152 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:19.421171 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:19.421215 (+ 44us) server_negotiation.cc:407] Connection header received
0926 13:44:19.422079 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:19.422102 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:19.422471 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:19.422847 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:19.423695 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:19.423701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:19.423716 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:19.423728 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:19.423791 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:19.423846 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:19.423976 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:19.424091 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 13:44:19.424243 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":146,"threads_started":1}
I20260926 13:44:19.425631 21739 log.cc:824] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:19.427894 21739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df: No bootstrap required, opened a new log
I20260926 13:44:19.428560 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:19.431995 21739 raft_consensus.cc:348] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42a0ba788a1449f925a84ad395764df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33039 } }
I20260926 13:44:19.432265 21739 raft_consensus.cc:374] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:19.432341 21739 raft_consensus.cc:729] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42a0ba788a1449f925a84ad395764df, State: Initialized, Role: FOLLOWER
I20260926 13:44:19.432902 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:19.430966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60291 (local address 127.0.0.1:33039)
0926 13:44:19.431116 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:19.431123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:19.431360 (+ 237us) server_negotiation.cc:407] Connection header received
0926 13:44:19.431467 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:19.431470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:19.431509 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:19.431594 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:19.432454 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:19.432458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:19.432461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:19.432474 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:19.432512 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:19.432524 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:19.432622 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:19.432718 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 13:44:19.432761 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 13:44:19.432857 21739 consensus_queue.cc:260] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42a0ba788a1449f925a84ad395764df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33039 } }
I20260926 13:44:19.433040 21739 raft_consensus.cc:388] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:19.433113 21739 raft_consensus.cc:482] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:19.433216 21739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:19.434460 21739 raft_consensus.cc:504] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42a0ba788a1449f925a84ad395764df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33039 } }
I20260926 13:44:19.434793 21739 leader_election.cc:302] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d42a0ba788a1449f925a84ad395764df; no voters:
I20260926 13:44:19.435102 21739 leader_election.cc:288] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:19.435175 21749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:19.435384 21749 raft_consensus.cc:686] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [term 1 LEADER]: Becoming Leader. State: Replica: d42a0ba788a1449f925a84ad395764df, State: Running, Role: LEADER
I20260926 13:44:19.435720 21749 consensus_queue.cc:237] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42a0ba788a1449f925a84ad395764df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33039 } }
I20260926 13:44:19.436547 21739 sys_catalog.cc:564] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:19.437515 21750 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [sys.catalog]: SysCatalogTable state changed. Reason: New leader d42a0ba788a1449f925a84ad395764df. Latest consensus state: current_term: 1 leader_uuid: "d42a0ba788a1449f925a84ad395764df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42a0ba788a1449f925a84ad395764df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33039 } } }
I20260926 13:44:19.437644 21750 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:19.437923 21751 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d42a0ba788a1449f925a84ad395764df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42a0ba788a1449f925a84ad395764df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33039 } } }
I20260926 13:44:19.438011 21751 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:19.438299 21758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:19.441520 21758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:19.448048 21758 catalog_manager.cc:1347] Generated new cluster ID: a229c522d1cf4fd78bb794b4aee79316
I20260926 13:44:19.448141 21758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:19.457621 21758 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:19.458045 21758 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:19.464457 21758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d42a0ba788a1449f925a84ad395764df: Generated new TSK 0
I20260926 13:44:19.465286 21758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:19.468891 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 21671
2026-09-26T13:44:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:19 dist-test-slave-1wns krb5kdc[21770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:19 dist-test-slave-1wns krb5kdc[21770](info): set up 2 sockets
Sep 26 13:44:19 dist-test-slave-1wns krb5kdc[21770](info): commencing operation
krb5kdc: starting...
W20260926 13:44:20.556279 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:20 dist-test-slave-1wns krb5kdc[21770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:44:20.687523 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:20.805828 21783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:20.806134 21783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:20.806180 21783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:20.810220 21783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:20.810297 21783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:20.810317 21783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:20.810334 21783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:20.810349 21783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:20.815941 21783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:20.817054 21783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:20.818065 21783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:20.826834 21788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:20.826853 21791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:20.826824 21789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:20.827332 21783 server_base.cc:1034] running on GCE node
I20260926 13:44:20.827962 21783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:20.829133 21783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:20.830312 21783 hybrid_clock.cc:648] HybridClock initialized: now 1790430260830283 us; error 43 us; skew 500 ppm
Sep 26 13:44:20 dist-test-slave-1wns krb5kdc[21770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:20.833331 21783 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:44:20.834270 21783 webserver.cc:460] Webserver started at http://127.0.0.1:40139/ using document root <none> and password file <none>
I20260926 13:44:20.834843 21783 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:20.834906 21783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:20.835120 21783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:20.837380 21783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "8eec706aaa62490387062f324f98dc9a"
format_stamp: "Formatted at 2026-09-26 13:44:20 on dist-test-slave-1wns"
server_key: "be7d94d177e9e93ea0e287e0e1a11808"
server_key_iv: "de4cd1caee5081dedf3cb0419ea9de76"
server_key_version: "encryptionkey@0"
I20260926 13:44:20.837980 21783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "8eec706aaa62490387062f324f98dc9a"
format_stamp: "Formatted at 2026-09-26 13:44:20 on dist-test-slave-1wns"
server_key: "be7d94d177e9e93ea0e287e0e1a11808"
server_key_iv: "de4cd1caee5081dedf3cb0419ea9de76"
server_key_version: "encryptionkey@0"
I20260926 13:44:20.841856 21783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:44:20.844513 21798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:20.845810 21783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:20.845953 21783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "8eec706aaa62490387062f324f98dc9a"
format_stamp: "Formatted at 2026-09-26 13:44:20 on dist-test-slave-1wns"
server_key: "be7d94d177e9e93ea0e287e0e1a11808"
server_key_iv: "de4cd1caee5081dedf3cb0419ea9de76"
server_key_version: "encryptionkey@0"
I20260926 13:44:20.846081 21783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:20.867650 21783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:20.871232 21783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:20.871450 21783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:20.878783 21783 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46307
I20260926 13:44:20.878808 21850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46307 every 8 connection(s)
I20260926 13:44:20.879774 21783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:20.882783 21851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:20.885026 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 21783
I20260926 13:44:20.885170 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:20.885473 1220 external_mini_cluster.cc:1428] Setting key 9457befb5dc3c3148ac8adcacb8b3222
I20260926 13:44:20.890264 21851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a: Bootstrap starting.
I20260926 13:44:20.893054 21851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:20.893931 21851 log.cc:824] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a: Log is configured to *not* fsync() on all Append() calls
Sep 26 13:44:20 dist-test-slave-1wns krb5kdc[21770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:44:20.896401 21851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a: No bootstrap required, opened a new log
I20260926 13:44:20.900722 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:20.886955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34915 (local address 127.0.0.1:46307)
0926 13:44:20.887389 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:20.887409 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:20.887453 (+ 44us) server_negotiation.cc:407] Connection header received
0926 13:44:20.888290 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:20.888319 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:20.888831 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:20.889228 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:20.889899 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:20.891314 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:20.892228 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:20.892915 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:20.893099 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:20.895937 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:20.895942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:20.895956 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:20.895982 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:20.897739 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:20.898261 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:20.898266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:20.898271 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:20.898379 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:20.898704 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:20.898706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:20.898707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:20.899097 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:20.899268 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:20.899588 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 13:44:20.899926 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":132,"threads_started":1}
I20260926 13:44:20.900702 21851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eec706aaa62490387062f324f98dc9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260926 13:44:20.901039 21851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:20.901104 21851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eec706aaa62490387062f324f98dc9a, State: Initialized, Role: FOLLOWER
I20260926 13:44:20.901607 21851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eec706aaa62490387062f324f98dc9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260926 13:44:20.901746 21851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:20.901808 21851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:20.901925 21851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:20.903009 21851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eec706aaa62490387062f324f98dc9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260926 13:44:20.903363 21851 leader_election.cc:302] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eec706aaa62490387062f324f98dc9a; no voters:
I20260926 13:44:20.903604 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:20.903702 21851 leader_election.cc:288] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:20.903787 21856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:20.904011 21856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [term 1 LEADER]: Becoming Leader. State: Replica: 8eec706aaa62490387062f324f98dc9a, State: Running, Role: LEADER
I20260926 13:44:20.904408 21856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eec706aaa62490387062f324f98dc9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260926 13:44:20.905092 21851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:20.906118 21857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8eec706aaa62490387062f324f98dc9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eec706aaa62490387062f324f98dc9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } } }
I20260926 13:44:20.906250 21857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:20.906481 21857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eec706aaa62490387062f324f98dc9a. Latest consensus state: current_term: 1 leader_uuid: "8eec706aaa62490387062f324f98dc9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eec706aaa62490387062f324f98dc9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } } }
I20260926 13:44:20.906567 21857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:20.906956 21870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:20.910364 21870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:20.914603 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:20.905871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36307 (local address 127.0.0.1:46307)
0926 13:44:20.906338 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:20.906344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:20.906603 (+ 259us) server_negotiation.cc:407] Connection header received
0926 13:44:20.906798 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:20.906801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:20.906850 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:20.906901 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:20.907863 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:20.908408 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:20.909285 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:20.909522 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:20.909627 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:20.911149 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:20.911151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:20.911153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:20.911175 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:20.912790 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:20.913288 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:20.913291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:20.913293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:20.913379 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:20.913675 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:20.913678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:20.913679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:20.913892 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:20.914035 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:20.914318 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 13:44:20.914442 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382}
I20260926 13:44:20.916596 21870 catalog_manager.cc:1347] Generated new cluster ID: e2707462eda84c13a945c96f937bd704
I20260926 13:44:20.916725 21870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:20.938753 21870 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:20.940083 21870 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:20.952099 21870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8eec706aaa62490387062f324f98dc9a: Generated new TSK 0
I20260926 13:44:20.953019 21870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:20.961380 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 21783
2026-09-26T13:44:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1493 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:21 dist-test-slave-1wns krb5kdc[21882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:21 dist-test-slave-1wns krb5kdc[21882](info): set up 2 sockets
Sep 26 13:44:21 dist-test-slave-1wns krb5kdc[21882](info): commencing operation
krb5kdc: starting...
W20260926 13:44:22.064577 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:22 dist-test-slave-1wns krb5kdc[21882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:44:22.203303 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:22.319352 21895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:22.319723 21895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:22.319864 21895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:22.324045 21895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:22.324156 21895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:22.324196 21895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:22.324234 21895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:22.324267 21895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:22.330003 21895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:22.331212 21895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:22.332381 21895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:22.341367 21900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:22.341384 21901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:22.341338 21903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:22.341746 21895 server_base.cc:1034] running on GCE node
I20260926 13:44:22.342360 21895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:22.343523 21895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:22.344705 21895 hybrid_clock.cc:648] HybridClock initialized: now 1790430262344682 us; error 40 us; skew 500 ppm
Sep 26 13:44:22 dist-test-slave-1wns krb5kdc[21882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:22.348004 21895 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:44:22.349007 21895 webserver.cc:460] Webserver started at http://127.0.0.1:33857/ using document root <none> and password file <none>
I20260926 13:44:22.349612 21895 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:22.349704 21895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:22.349936 21895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:22.352138 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "f22b59409e1345579c59e44d1374cb65"
format_stamp: "Formatted at 2026-09-26 13:44:22 on dist-test-slave-1wns"
server_key: "8b9dd34f6781a30d37032fd8ceaf5eba"
server_key_iv: "3654c0edbb7c63ae9c5c9d901584fdc7"
server_key_version: "encryptionkey@0"
I20260926 13:44:22.352672 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "f22b59409e1345579c59e44d1374cb65"
format_stamp: "Formatted at 2026-09-26 13:44:22 on dist-test-slave-1wns"
server_key: "8b9dd34f6781a30d37032fd8ceaf5eba"
server_key_iv: "3654c0edbb7c63ae9c5c9d901584fdc7"
server_key_version: "encryptionkey@0"
I20260926 13:44:22.356549 21895 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 13:44:22.359159 21910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:22.360875 21895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 13:44:22.361042 21895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "f22b59409e1345579c59e44d1374cb65"
format_stamp: "Formatted at 2026-09-26 13:44:22 on dist-test-slave-1wns"
server_key: "8b9dd34f6781a30d37032fd8ceaf5eba"
server_key_iv: "3654c0edbb7c63ae9c5c9d901584fdc7"
server_key_version: "encryptionkey@0"
I20260926 13:44:22.361182 21895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:22.375226 21895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:22.380034 21895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:22.380298 21895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:22.388967 21895 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45181
I20260926 13:44:22.388993 21962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45181 every 8 connection(s)
I20260926 13:44:22.390023 21895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:22.390666 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 21895
I20260926 13:44:22.390836 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:22.391158 1220 external_mini_cluster.cc:1428] Setting key a1b7f9654dab89271d2905f2e4857490
I20260926 13:44:22.393400 21963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:22.401614 21963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65: Bootstrap starting.
I20260926 13:44:22.402639 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:22.392775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40171 (local address 127.0.0.1:45181)
0926 13:44:22.393219 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:22.393238 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:22.393283 (+ 45us) server_negotiation.cc:407] Connection header received
0926 13:44:22.394145 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:22.394169 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:22.394628 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:22.395152 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:22.395601 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:22.397260 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:22.398302 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:22.399085 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:22.399301 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:22.400352 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:22.400364 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:22.400385 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:22.400972 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:22.401049 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:22.401081 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:22.401190 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:22.401380 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 13:44:22.401623 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":126,"threads_started":1}
I20260926 13:44:22.404554 21963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:22.405568 21963 log.cc:824] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:22.406359 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:22.408214 21963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65: No bootstrap required, opened a new log
I20260926 13:44:22.412953 21963 raft_consensus.cc:348] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22b59409e1345579c59e44d1374cb65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45181 } }
I20260926 13:44:22.413281 21963 raft_consensus.cc:374] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:22.413404 21963 raft_consensus.cc:729] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f22b59409e1345579c59e44d1374cb65, State: Initialized, Role: FOLLOWER
I20260926 13:44:22.413564 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:22.409055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47417 (local address 127.0.0.1:45181)
0926 13:44:22.409182 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:22.409199 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:22.409301 (+ 102us) server_negotiation.cc:407] Connection header received
0926 13:44:22.409486 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:22.409495 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:22.409543 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:22.409644 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:22.409954 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:22.410494 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:22.411391 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:22.411637 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:22.411791 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:22.412970 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:22.412973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:22.412975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:22.412991 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:22.413021 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:22.413034 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:22.413129 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:22.413256 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 13:44:22.413322 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 13:44:22.414014 21963 consensus_queue.cc:260] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22b59409e1345579c59e44d1374cb65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45181 } }
I20260926 13:44:22.414196 21963 raft_consensus.cc:388] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:22.414271 21963 raft_consensus.cc:482] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:22.414377 21963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:22.415552 21963 raft_consensus.cc:504] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22b59409e1345579c59e44d1374cb65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45181 } }
I20260926 13:44:22.415971 21963 leader_election.cc:302] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f22b59409e1345579c59e44d1374cb65; no voters:
I20260926 13:44:22.416308 21963 leader_election.cc:288] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:22.416555 21973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:22.416947 21973 raft_consensus.cc:686] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [term 1 LEADER]: Becoming Leader. State: Replica: f22b59409e1345579c59e44d1374cb65, State: Running, Role: LEADER
I20260926 13:44:22.417465 21973 consensus_queue.cc:237] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22b59409e1345579c59e44d1374cb65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45181 } }
I20260926 13:44:22.418071 21963 sys_catalog.cc:564] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:22.419268 21975 sys_catalog.cc:455] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f22b59409e1345579c59e44d1374cb65. Latest consensus state: current_term: 1 leader_uuid: "f22b59409e1345579c59e44d1374cb65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22b59409e1345579c59e44d1374cb65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45181 } } }
I20260926 13:44:22.419414 21975 sys_catalog.cc:458] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:22.419736 21974 sys_catalog.cc:455] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f22b59409e1345579c59e44d1374cb65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22b59409e1345579c59e44d1374cb65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45181 } } }
I20260926 13:44:22.419871 21982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:22.419932 21974 sys_catalog.cc:458] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:22.423537 21982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:22.430454 21982 catalog_manager.cc:1347] Generated new cluster ID: 7a6a6d7b21904e97a7f3b358abb99eef
I20260926 13:44:22.430613 21982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:22.443849 21982 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:22.444878 21982 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:22.458169 21982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f22b59409e1345579c59e44d1374cb65: Generated new TSK 0
I20260926 13:44:22.458995 21982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:22.465456 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 21895
2026-09-26T13:44:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1504 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:22 dist-test-slave-1wns krb5kdc[21994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:22 dist-test-slave-1wns krb5kdc[21994](info): set up 2 sockets
Sep 26 13:44:22 dist-test-slave-1wns krb5kdc[21994](info): commencing operation
krb5kdc: starting...
W20260926 13:44:23.556169 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:23 dist-test-slave-1wns krb5kdc[21994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:44:23.691987 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:23.814258 22007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:23.814611 22007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:23.814694 22007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:23.819504 22007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:23.819623 22007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:23.819679 22007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:23.819720 22007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:23.819754 22007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:23.826116 22007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:23.827397 22007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:23.828701 22007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:23.838531 22013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:23.838749 22015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:23.838728 22012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:23.839123 22007 server_base.cc:1034] running on GCE node
I20260926 13:44:23.839884 22007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:23.841102 22007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:23.842319 22007 hybrid_clock.cc:648] HybridClock initialized: now 1790430263842287 us; error 47 us; skew 500 ppm
Sep 26 13:44:23 dist-test-slave-1wns krb5kdc[21994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:23.845721 22007 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:44:23.846844 22007 webserver.cc:460] Webserver started at http://127.0.0.1:44315/ using document root <none> and password file <none>
I20260926 13:44:23.847456 22007 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:23.847523 22007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:23.847748 22007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:23.851060 22007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "62cbcbeb4a574b8a9465725098997bb3"
format_stamp: "Formatted at 2026-09-26 13:44:23 on dist-test-slave-1wns"
server_key: "66d4bfab02b362c9c276f129ac319228"
server_key_iv: "1579cb3603c92a97702d913b49aaa592"
server_key_version: "encryptionkey@0"
I20260926 13:44:23.851913 22007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "62cbcbeb4a574b8a9465725098997bb3"
format_stamp: "Formatted at 2026-09-26 13:44:23 on dist-test-slave-1wns"
server_key: "66d4bfab02b362c9c276f129ac319228"
server_key_iv: "1579cb3603c92a97702d913b49aaa592"
server_key_version: "encryptionkey@0"
I20260926 13:44:23.857945 22007 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 13:44:23.862175 22022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:23.864097 22007 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 13:44:23.864277 22007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "62cbcbeb4a574b8a9465725098997bb3"
format_stamp: "Formatted at 2026-09-26 13:44:23 on dist-test-slave-1wns"
server_key: "66d4bfab02b362c9c276f129ac319228"
server_key_iv: "1579cb3603c92a97702d913b49aaa592"
server_key_version: "encryptionkey@0"
I20260926 13:44:23.864432 22007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:23.871716 22007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:23.876163 22007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:23.876444 22007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:23.886093 22007 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35815
I20260926 13:44:23.886518 22074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35815 every 8 connection(s)
I20260926 13:44:23.887398 22007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:23.889624 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 22007
I20260926 13:44:23.889788 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:23.890264 1220 external_mini_cluster.cc:1428] Setting key 4cfe9581289948e3e85cdb03861bb802
I20260926 13:44:23.891489 22075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:23.899133 22078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:23.891939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60289 (local address 127.0.0.1:35815)
0926 13:44:23.893527 (+ 1588us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:23.893566 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:23.893616 (+ 50us) server_negotiation.cc:407] Connection header received
0926 13:44:23.895364 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:23.895396 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:23.895898 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:23.896336 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:23.897189 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:23.897197 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:23.897215 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:23.897229 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:23.897297 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:23.897326 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:23.897830 (+ 504us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:23.897910 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 13:44:23.898072 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1359,"thread_start_us":152,"threads_started":1}
I20260926 13:44:23.901890 22075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3: Bootstrap starting.
I20260926 13:44:23.902432 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:23.905447 22075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:23.906533 22075 log.cc:824] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:23.906778 22078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:23.904906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37877 (local address 127.0.0.1:35815)
0926 13:44:23.905010 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:23.905014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:23.905324 (+ 310us) server_negotiation.cc:407] Connection header received
0926 13:44:23.905478 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:23.905480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:23.905528 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:23.905600 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:23.906373 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:23.906377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:23.906380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:23.906391 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:23.906421 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:23.906433 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:23.906515 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:23.906606 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 13:44:23.906647 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:44:23.909336 22075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3: No bootstrap required, opened a new log
I20260926 13:44:23.915477 22075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62cbcbeb4a574b8a9465725098997bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } }
I20260926 13:44:23.915943 22075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:23.916059 22075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62cbcbeb4a574b8a9465725098997bb3, State: Initialized, Role: FOLLOWER
I20260926 13:44:23.916666 22075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62cbcbeb4a574b8a9465725098997bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } }
I20260926 13:44:23.916896 22075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:23.917006 22075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:23.917155 22075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:23.918712 22075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62cbcbeb4a574b8a9465725098997bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } }
I20260926 13:44:23.919215 22075 leader_election.cc:302] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62cbcbeb4a574b8a9465725098997bb3; no voters:
I20260926 13:44:23.919678 22075 leader_election.cc:288] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:23.919790 22085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:23.920064 22085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [term 1 LEADER]: Becoming Leader. State: Replica: 62cbcbeb4a574b8a9465725098997bb3, State: Running, Role: LEADER
I20260926 13:44:23.920401 22085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62cbcbeb4a574b8a9465725098997bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } }
I20260926 13:44:23.921398 22075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:23.922060 22086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62cbcbeb4a574b8a9465725098997bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62cbcbeb4a574b8a9465725098997bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } } }
I20260926 13:44:23.922194 22086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:23.922457 22087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62cbcbeb4a574b8a9465725098997bb3. Latest consensus state: current_term: 1 leader_uuid: "62cbcbeb4a574b8a9465725098997bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62cbcbeb4a574b8a9465725098997bb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } } }
I20260926 13:44:23.922562 22087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:23.922706 22091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:23.925748 22091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:23.931685 22091 catalog_manager.cc:1347] Generated new cluster ID: faae85f198f740c0af5bd37163187db9
I20260926 13:44:23.931759 22091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:23.940199 22091 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:23.940585 22091 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:23.948724 22091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62cbcbeb4a574b8a9465725098997bb3: Generated new TSK 0
I20260926 13:44:23.949620 22091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:23.960904 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 22007
2026-09-26T13:44:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1495 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:24 dist-test-slave-1wns krb5kdc[22106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:24 dist-test-slave-1wns krb5kdc[22106](info): set up 2 sockets
Sep 26 13:44:24 dist-test-slave-1wns krb5kdc[22106](info): commencing operation
krb5kdc: starting...
W20260926 13:44:25.048501 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:25 dist-test-slave-1wns krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:44:25.184513 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33103
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:25.313503 22119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:25.313863 22119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:25.313966 22119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:25.319271 22119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:25.319383 22119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:25.319406 22119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:25.319422 22119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:25.319439 22119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:25.325599 22119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:25.326861 22119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:25.328140 22119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:25.338070 22124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:25.338370 22119 server_base.cc:1034] running on GCE node
W20260926 13:44:25.338070 22127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:25.338073 22125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:25.339161 22119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:25.340623 22119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:25.341871 22119 hybrid_clock.cc:648] HybridClock initialized: now 1790430265341854 us; error 32 us; skew 500 ppm
Sep 26 13:44:25 dist-test-slave-1wns krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:25.351699 22119 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:44:25.352821 22119 webserver.cc:460] Webserver started at http://127.0.0.1:34271/ using document root <none> and password file <none>
I20260926 13:44:25.353442 22119 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:25.353511 22119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:25.353740 22119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:25.356138 22119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "7511a558ef6d40ee8e1576da52e90d74"
format_stamp: "Formatted at 2026-09-26 13:44:25 on dist-test-slave-1wns"
server_key: "554299a95e5f9a9759c00523a08dca8d"
server_key_iv: "8b37c17212fbaa70e2cb277b2f2e7dc0"
server_key_version: "encryptionkey@0"
I20260926 13:44:25.356792 22119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "7511a558ef6d40ee8e1576da52e90d74"
format_stamp: "Formatted at 2026-09-26 13:44:25 on dist-test-slave-1wns"
server_key: "554299a95e5f9a9759c00523a08dca8d"
server_key_iv: "8b37c17212fbaa70e2cb277b2f2e7dc0"
server_key_version: "encryptionkey@0"
I20260926 13:44:25.361037 22119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:44:25.363993 22134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:25.365551 22119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:25.365754 22119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "7511a558ef6d40ee8e1576da52e90d74"
format_stamp: "Formatted at 2026-09-26 13:44:25 on dist-test-slave-1wns"
server_key: "554299a95e5f9a9759c00523a08dca8d"
server_key_iv: "8b37c17212fbaa70e2cb277b2f2e7dc0"
server_key_version: "encryptionkey@0"
I20260926 13:44:25.365906 22119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:25.398011 22119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:25.401886 22119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:25.402078 22119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:25.409806 22119 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33103
I20260926 13:44:25.409829 22186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33103 every 8 connection(s)
I20260926 13:44:25.410887 22119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:25.412678 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 22119
I20260926 13:44:25.412791 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:25.413134 1220 external_mini_cluster.cc:1428] Setting key 7f68b3837475b0bd73ea2f098aa7e0a7
I20260926 13:44:25.414548 22187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:25.422366 22187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74: Bootstrap starting.
Sep 26 13:44:25 dist-test-slave-1wns krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:44:25.425349 22187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:25.426213 22187 log.cc:824] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:25.428938 22187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74: No bootstrap required, opened a new log
I20260926 13:44:25.430045 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:25.414750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:44425 (local address 127.0.0.1:33103)
0926 13:44:25.415195 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:25.415217 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:25.415254 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:44:25.416033 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:25.416053 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:25.416459 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:25.416808 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:25.417937 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:25.419207 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:25.420671 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:25.421244 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:25.421390 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:25.424612 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:25.424618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:25.424632 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:25.424666 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:25.426684 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:25.427372 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:25.427378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:25.427383 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:25.427515 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:25.427945 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:25.427948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:25.427949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:25.428396 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:25.428597 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:25.428960 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 13:44:25.429245 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":151,"threads_started":1}
I20260926 13:44:25.433948 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:25.433972 22187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7511a558ef6d40ee8e1576da52e90d74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33103 } }
I20260926 13:44:25.434324 22187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:25.434410 22187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7511a558ef6d40ee8e1576da52e90d74, State: Initialized, Role: FOLLOWER
I20260926 13:44:25.434923 22187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7511a558ef6d40ee8e1576da52e90d74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33103 } }
I20260926 13:44:25.435098 22187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:25.435235 22187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:25.435366 22187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:25.436805 22187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7511a558ef6d40ee8e1576da52e90d74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33103 } }
I20260926 13:44:25.437225 22187 leader_election.cc:302] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7511a558ef6d40ee8e1576da52e90d74; no voters:
I20260926 13:44:25.437582 22187 leader_election.cc:288] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:25.437716 22197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:25.437987 22197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [term 1 LEADER]: Becoming Leader. State: Replica: 7511a558ef6d40ee8e1576da52e90d74, State: Running, Role: LEADER
I20260926 13:44:25.438407 22197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7511a558ef6d40ee8e1576da52e90d74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33103 } }
I20260926 13:44:25.439155 22187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:25.439749 22197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7511a558ef6d40ee8e1576da52e90d74. Latest consensus state: current_term: 1 leader_uuid: "7511a558ef6d40ee8e1576da52e90d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7511a558ef6d40ee8e1576da52e90d74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33103 } } }
I20260926 13:44:25.440853 22197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:25.440717 22198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7511a558ef6d40ee8e1576da52e90d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7511a558ef6d40ee8e1576da52e90d74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33103 } } }
I20260926 13:44:25.444195 22198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74 [sys.catalog]: This master's current role is: LEADER
W20260926 13:44:25.445533 22212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:25.445675 22212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:25.445785 22211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:25.446980 22211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:25.447212 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:25.436249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:40559 (local address 127.0.0.1:33103)
0926 13:44:25.436429 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:25.436433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:25.436668 (+ 235us) server_negotiation.cc:407] Connection header received
0926 13:44:25.436755 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:25.436758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:25.436824 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:25.436924 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:25.437807 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:25.438661 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:25.439869 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:25.440230 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:25.440332 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:25.442482 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:25.442485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:25.442495 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:25.442522 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:25.444713 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:25.445515 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:25.445519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:25.445521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:25.445729 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:25.446144 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:25.446148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:25.446150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:25.446399 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:25.446564 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:25.446937 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 13:44:25.447053 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 13:44:25.453927 22211 catalog_manager.cc:1347] Generated new cluster ID: 1d1a56313aee49468409f4adac5bd1ff
I20260926 13:44:25.454103 22211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:25.495837 22211 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:25.496897 22211 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:25.507011 22211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7511a558ef6d40ee8e1576da52e90d74: Generated new TSK 0
I20260926 13:44:25.507884 22211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:25.517831 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 22119
W20260926 13:44:25.523532 22104 connection.cc:535] client connection to 127.0.0.1:33103 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33103): Connection reset by peer (error 104)
2026-09-26T13:44:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1556 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:25 dist-test-slave-1wns krb5kdc[22218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:25 dist-test-slave-1wns krb5kdc[22218](info): set up 2 sockets
Sep 26 13:44:25 dist-test-slave-1wns krb5kdc[22218](info): commencing operation
krb5kdc: starting...
W20260926 13:44:26.576146 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:26 dist-test-slave-1wns krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:44:26.710788 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37923
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:26.830092 22231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:26.830440 22231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:26.830546 22231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:26.834802 22231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:26.834916 22231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:26.834955 22231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:26.834995 22231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:26.835052 22231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:26.840888 22231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:26.842132 22231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:26.843264 22231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:26.852666 22239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:26.852723 22236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:26.852843 22231 server_base.cc:1034] running on GCE node
W20260926 13:44:26.852689 22237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:26.853824 22231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:26.854950 22231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:26.856113 22231 hybrid_clock.cc:648] HybridClock initialized: now 1790430266856072 us; error 50 us; skew 500 ppm
Sep 26 13:44:26 dist-test-slave-1wns krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:26.859092 22231 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:44:26.860271 22231 webserver.cc:460] Webserver started at http://127.0.0.1:36777/ using document root <none> and password file <none>
I20260926 13:44:26.860853 22231 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:26.860916 22231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:26.861177 22231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:26.863312 22231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "70d8606998a649f2b0e493e63a5fd55c"
format_stamp: "Formatted at 2026-09-26 13:44:26 on dist-test-slave-1wns"
server_key: "030f59492d5aa48cdb457a5ea01cffc3"
server_key_iv: "c4f5e46af9e508fffa7e3e9f61c42954"
server_key_version: "encryptionkey@0"
I20260926 13:44:26.863968 22231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "70d8606998a649f2b0e493e63a5fd55c"
format_stamp: "Formatted at 2026-09-26 13:44:26 on dist-test-slave-1wns"
server_key: "030f59492d5aa48cdb457a5ea01cffc3"
server_key_iv: "c4f5e46af9e508fffa7e3e9f61c42954"
server_key_version: "encryptionkey@0"
I20260926 13:44:26.867760 22231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:44:26.870361 22246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:26.871671 22231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:44:26.871803 22231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "70d8606998a649f2b0e493e63a5fd55c"
format_stamp: "Formatted at 2026-09-26 13:44:26 on dist-test-slave-1wns"
server_key: "030f59492d5aa48cdb457a5ea01cffc3"
server_key_iv: "c4f5e46af9e508fffa7e3e9f61c42954"
server_key_version: "encryptionkey@0"
I20260926 13:44:26.871938 22231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:26.884859 22231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:26.888338 22231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:26.888551 22231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:26.895862 22231 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37923
I20260926 13:44:26.895879 22298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37923 every 8 connection(s)
I20260926 13:44:26.896855 22231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:26.898304 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 22231
I20260926 13:44:26.898416 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:26.898731 1220 external_mini_cluster.cc:1428] Setting key 2925736307708ea6f16f50748a36d5e9
I20260926 13:44:26.900319 22299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:26.908360 22299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c: Bootstrap starting.
I20260926 13:44:26.910195 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:26.900353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:58077 (local address 127.0.0.1:37923)
0926 13:44:26.900832 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:26.900854 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:26.900898 (+ 44us) server_negotiation.cc:407] Connection header received
0926 13:44:26.901827 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:26.901851 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:26.902339 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:26.902743 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:26.903198 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:26.904902 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:26.905911 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:26.906725 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:26.906900 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:26.907838 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:26.907848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:26.907868 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:26.908441 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:26.908525 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:26.908559 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:26.908704 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:26.908960 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 13:44:26.909124 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":135,"threads_started":1}
I20260926 13:44:26.911355 22299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:26.912251 22299 log.cc:824] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:26.913697 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:26.914774 22299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c: No bootstrap required, opened a new log
I20260926 13:44:26.919612 22299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d8606998a649f2b0e493e63a5fd55c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37923 } }
I20260926 13:44:26.919958 22299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:26.920045 22299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d8606998a649f2b0e493e63a5fd55c, State: Initialized, Role: FOLLOWER
I20260926 13:44:26.920495 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:26.916111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:58043 (local address 127.0.0.1:37923)
0926 13:44:26.916260 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:26.916265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:26.916512 (+ 247us) server_negotiation.cc:407] Connection header received
0926 13:44:26.916646 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:26.916648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:26.916690 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:26.916756 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:26.917042 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:26.917564 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:26.918711 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:26.918919 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:26.918994 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:26.919910 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:26.919913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:26.919915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:26.919928 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:26.919978 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:26.919995 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:26.920111 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:26.920303 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 13:44:26.920343 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:44:26.920596 22299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d8606998a649f2b0e493e63a5fd55c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37923 } }
I20260926 13:44:26.920765 22299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:26.920832 22299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:26.920933 22299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:26.921974 22299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d8606998a649f2b0e493e63a5fd55c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37923 } }
I20260926 13:44:26.922276 22299 leader_election.cc:302] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70d8606998a649f2b0e493e63a5fd55c; no voters:
I20260926 13:44:26.922539 22299 leader_election.cc:288] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:26.922624 22309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:26.922876 22309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [term 1 LEADER]: Becoming Leader. State: Replica: 70d8606998a649f2b0e493e63a5fd55c, State: Running, Role: LEADER
I20260926 13:44:26.923298 22309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d8606998a649f2b0e493e63a5fd55c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37923 } }
I20260926 13:44:26.923920 22299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:26.925069 22310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70d8606998a649f2b0e493e63a5fd55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d8606998a649f2b0e493e63a5fd55c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37923 } } }
I20260926 13:44:26.925208 22310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:26.925494 22311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d8606998a649f2b0e493e63a5fd55c. Latest consensus state: current_term: 1 leader_uuid: "70d8606998a649f2b0e493e63a5fd55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d8606998a649f2b0e493e63a5fd55c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37923 } } }
I20260926 13:44:26.925590 22311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:26.925597 22318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:26.929255 22318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:26.934890 22318 catalog_manager.cc:1347] Generated new cluster ID: 30c2656a17ff43f0928e5d55490776e8
I20260926 13:44:26.934999 22318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:26.953392 22318 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:26.954780 22318 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:26.962973 22318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70d8606998a649f2b0e493e63a5fd55c: Generated new TSK 0
I20260926 13:44:26.963974 22318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:26.974205 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 22231
W20260926 13:44:26.980304 22216 connection.cc:535] client connection to 127.0.0.1:37923 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37923): Connection reset by peer (error 104)
2026-09-26T13:44:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:27 dist-test-slave-1wns krb5kdc[22330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:27 dist-test-slave-1wns krb5kdc[22330](info): set up 2 sockets
Sep 26 13:44:27 dist-test-slave-1wns krb5kdc[22330](info): commencing operation
krb5kdc: starting...
W20260926 13:44:28.080287 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:28 dist-test-slave-1wns krb5kdc[22330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:44:28.214932 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:28.331096 22343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:28.331393 22343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:28.331436 22343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:28.335644 22343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:28.335737 22343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:28.335758 22343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:28.335775 22343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:28.335790 22343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:28.341604 22343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:28.342810 22343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:28.344002 22343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:28.353089 22348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:28.353089 22351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:28.353104 22349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:28.353166 22343 server_base.cc:1034] running on GCE node
I20260926 13:44:28.353987 22343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:28.355302 22343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:28.356509 22343 hybrid_clock.cc:648] HybridClock initialized: now 1790430268356500 us; error 34 us; skew 500 ppm
Sep 26 13:44:28 dist-test-slave-1wns krb5kdc[22330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:28.360078 22343 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:44:28.361289 22343 webserver.cc:460] Webserver started at http://127.0.0.1:33723/ using document root <none> and password file <none>
I20260926 13:44:28.361949 22343 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:28.362052 22343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:28.362317 22343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:28.364789 22343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "f5eb98d124764ed39e9ab2d0a3e83791"
format_stamp: "Formatted at 2026-09-26 13:44:28 on dist-test-slave-1wns"
server_key: "a6a81bde4bf44a9e7dc64bd480c448dc"
server_key_iv: "f57afea40689007168b9a576fb5e90bb"
server_key_version: "encryptionkey@0"
I20260926 13:44:28.365465 22343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "f5eb98d124764ed39e9ab2d0a3e83791"
format_stamp: "Formatted at 2026-09-26 13:44:28 on dist-test-slave-1wns"
server_key: "a6a81bde4bf44a9e7dc64bd480c448dc"
server_key_iv: "f57afea40689007168b9a576fb5e90bb"
server_key_version: "encryptionkey@0"
I20260926 13:44:28.369849 22343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 13:44:28.373027 22358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:28.374423 22343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 13:44:28.374611 22343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "f5eb98d124764ed39e9ab2d0a3e83791"
format_stamp: "Formatted at 2026-09-26 13:44:28 on dist-test-slave-1wns"
server_key: "a6a81bde4bf44a9e7dc64bd480c448dc"
server_key_iv: "f57afea40689007168b9a576fb5e90bb"
server_key_version: "encryptionkey@0"
I20260926 13:44:28.374758 22343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:28.380230 22343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:28.383764 22343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:28.384018 22343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:28.391409 22343 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46279
I20260926 13:44:28.391454 22410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46279 every 8 connection(s)
I20260926 13:44:28.392582 22343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:28.395589 22411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:28.401459 22411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791: Bootstrap starting.
I20260926 13:44:28.402029 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 22343
I20260926 13:44:28.402289 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:28.402642 1220 external_mini_cluster.cc:1428] Setting key 8c8231f461de60b457ec61feaaee62f6
I20260926 13:44:28.404631 22411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:28.405562 22411 log.cc:824] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:28.408165 22411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791: No bootstrap required, opened a new log
I20260926 13:44:28.408933 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:28.404140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:60241 (local address 127.0.0.1:46279)
0926 13:44:28.404626 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:28.404647 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:28.404694 (+ 47us) server_negotiation.cc:407] Connection header received
0926 13:44:28.405644 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:28.405672 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:28.406080 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:28.406498 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:28.407338 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:28.407344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:28.407359 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:28.407371 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:28.407436 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:28.407466 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:28.407584 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:28.407753 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 13:44:28.407937 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":125,"threads_started":1}
I20260926 13:44:28.412485 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:28.412688 22411 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5eb98d124764ed39e9ab2d0a3e83791" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46279 } }
I20260926 13:44:28.412932 22411 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:28.413015 22411 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5eb98d124764ed39e9ab2d0a3e83791, State: Initialized, Role: FOLLOWER
I20260926 13:44:28.413512 22411 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5eb98d124764ed39e9ab2d0a3e83791" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46279 } }
I20260926 13:44:28.413640 22411 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:28.413729 22411 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:28.413852 22411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:28.414958 22411 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5eb98d124764ed39e9ab2d0a3e83791" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46279 } }
I20260926 13:44:28.415350 22411 leader_election.cc:302] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5eb98d124764ed39e9ab2d0a3e83791; no voters:
I20260926 13:44:28.415743 22411 leader_election.cc:288] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:28.415829 22421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:28.416075 22421 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [term 1 LEADER]: Becoming Leader. State: Replica: f5eb98d124764ed39e9ab2d0a3e83791, State: Running, Role: LEADER
I20260926 13:44:28.416405 22421 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5eb98d124764ed39e9ab2d0a3e83791" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46279 } }
I20260926 13:44:28.417203 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:28.415040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:43213 (local address 127.0.0.1:46279)
0926 13:44:28.415163 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:28.415168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:28.415442 (+ 274us) server_negotiation.cc:407] Connection header received
0926 13:44:28.415620 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:28.415632 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:28.415680 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:28.415774 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:28.416709 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:28.416713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:28.416715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:28.416728 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:28.416764 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:28.416780 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:28.416845 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:28.417005 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 13:44:28.417053 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:44:28.417238 22411 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:28.418669 22422 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5eb98d124764ed39e9ab2d0a3e83791. Latest consensus state: current_term: 1 leader_uuid: "f5eb98d124764ed39e9ab2d0a3e83791" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5eb98d124764ed39e9ab2d0a3e83791" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46279 } } }
I20260926 13:44:28.418809 22422 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:28.419104 22423 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5eb98d124764ed39e9ab2d0a3e83791" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5eb98d124764ed39e9ab2d0a3e83791" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46279 } } }
I20260926 13:44:28.419262 22423 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791 [sys.catalog]: This master's current role is: LEADER
W20260926 13:44:28.422749 22436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:28.422866 22436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:28.422941 22431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:28.423691 22431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:28.430076 22431 catalog_manager.cc:1347] Generated new cluster ID: 53a92432c7f5403dadaeea88b0370c82
I20260926 13:44:28.430178 22431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:28.442174 22431 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:28.442580 22431 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:28.461478 22431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5eb98d124764ed39e9ab2d0a3e83791: Generated new TSK 0
I20260926 13:44:28.462618 22431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:28.476837 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 22343
2026-09-26T13:44:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1501 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:28 dist-test-slave-1wns krb5kdc[22442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:28 dist-test-slave-1wns krb5kdc[22442](info): set up 2 sockets
Sep 26 13:44:28 dist-test-slave-1wns krb5kdc[22442](info): commencing operation
krb5kdc: starting...
W20260926 13:44:29.544678 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:29 dist-test-slave-1wns krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:44:29.671576 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:29.784730 22455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:29.785053 22455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:29.785130 22455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:29.789496 22455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:29.789597 22455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:29.789634 22455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:29.789693 22455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:29.789748 22455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:29.795548 22455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:29.796784 22455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:29.797850 22455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:29.806922 22461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:29.806922 22460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:29.807005 22463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:29.807663 22455 server_base.cc:1034] running on GCE node
I20260926 13:44:29.808324 22455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:29.809604 22455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:29.810796 22455 hybrid_clock.cc:648] HybridClock initialized: now 1790430269810769 us; error 51 us; skew 500 ppm
Sep 26 13:44:29 dist-test-slave-1wns krb5kdc[22442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:29.813777 22455 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:44:29.814713 22455 webserver.cc:460] Webserver started at http://127.0.0.1:46755/ using document root <none> and password file <none>
I20260926 13:44:29.815339 22455 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:29.815440 22455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:29.815722 22455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:29.818707 22455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "f721449ddc2645f5bfa3d196dd99d88d"
format_stamp: "Formatted at 2026-09-26 13:44:29 on dist-test-slave-1wns"
server_key: "fa99a5960398c1bdd3017e16105e7527"
server_key_iv: "babc6e505f2665f60f1d1c48c2a03e02"
server_key_version: "encryptionkey@0"
I20260926 13:44:29.819510 22455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "f721449ddc2645f5bfa3d196dd99d88d"
format_stamp: "Formatted at 2026-09-26 13:44:29 on dist-test-slave-1wns"
server_key: "fa99a5960398c1bdd3017e16105e7527"
server_key_iv: "babc6e505f2665f60f1d1c48c2a03e02"
server_key_version: "encryptionkey@0"
I20260926 13:44:29.824949 22455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 13:44:29.827879 22470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:29.829519 22455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:29.829690 22455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "f721449ddc2645f5bfa3d196dd99d88d"
format_stamp: "Formatted at 2026-09-26 13:44:29 on dist-test-slave-1wns"
server_key: "fa99a5960398c1bdd3017e16105e7527"
server_key_iv: "babc6e505f2665f60f1d1c48c2a03e02"
server_key_version: "encryptionkey@0"
I20260926 13:44:29.829833 22455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:29.858091 22455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:29.861802 22455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:29.862020 22455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:29.869799 22455 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45055
I20260926 13:44:29.869822 22522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45055 every 8 connection(s)
I20260926 13:44:29.870798 22455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:29.873777 22523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:29.879017 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 22455
I20260926 13:44:29.879191 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:29.879467 1220 external_mini_cluster.cc:1428] Setting key d0b38fbc29b2eb97f92b543c3a745f0d
I20260926 13:44:29.879451 22523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d: Bootstrap starting.
I20260926 13:44:29.882133 22523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:29.882889 22523 log.cc:824] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:29.884905 22523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d: No bootstrap required, opened a new log
I20260926 13:44:29.888492 22523 raft_consensus.cc:348] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f721449ddc2645f5bfa3d196dd99d88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45055 } }
I20260926 13:44:29.888846 22523 raft_consensus.cc:374] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:29.888909 22523 raft_consensus.cc:729] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f721449ddc2645f5bfa3d196dd99d88d, State: Initialized, Role: FOLLOWER
Sep 26 13:44:29 dist-test-slave-1wns krb5kdc[22442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:44:29.889473 22523 consensus_queue.cc:260] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f721449ddc2645f5bfa3d196dd99d88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45055 } }
I20260926 13:44:29.889671 22523 raft_consensus.cc:388] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:29.889746 22523 raft_consensus.cc:482] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:29.889855 22523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:29.891153 22523 raft_consensus.cc:504] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f721449ddc2645f5bfa3d196dd99d88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45055 } }
I20260926 13:44:29.891589 22523 leader_election.cc:302] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f721449ddc2645f5bfa3d196dd99d88d; no voters:
I20260926 13:44:29.892019 22523 leader_election.cc:288] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:29.892154 22528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:29.892539 22528 raft_consensus.cc:686] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [term 1 LEADER]: Becoming Leader. State: Replica: f721449ddc2645f5bfa3d196dd99d88d, State: Running, Role: LEADER
I20260926 13:44:29.892926 22528 consensus_queue.cc:237] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f721449ddc2645f5bfa3d196dd99d88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45055 } }
I20260926 13:44:29.893496 22523 sys_catalog.cc:564] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:29.895787 22530 sys_catalog.cc:455] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f721449ddc2645f5bfa3d196dd99d88d. Latest consensus state: current_term: 1 leader_uuid: "f721449ddc2645f5bfa3d196dd99d88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f721449ddc2645f5bfa3d196dd99d88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45055 } } }
I20260926 13:44:29.896723 22530 sys_catalog.cc:458] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:29.895792 22529 sys_catalog.cc:455] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f721449ddc2645f5bfa3d196dd99d88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f721449ddc2645f5bfa3d196dd99d88d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45055 } } }
I20260926 13:44:29.897251 22529 sys_catalog.cc:458] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:29.896421 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:29.880968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:45185 (local address 127.0.0.1:45055)
0926 13:44:29.881421 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:29.881444 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:29.881482 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:44:29.882228 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:29.882278 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:29.882687 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:29.883054 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:29.883992 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:29.885131 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:29.886111 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:29.886641 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:29.886833 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:29.890044 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:29.890053 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:29.890073 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:29.890113 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:29.892360 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:29.893032 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:29.893038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:29.893045 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:29.893218 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:29.893614 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:29.893618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:29.893619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:29.894176 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:29.894409 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:29.894935 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 13:44:29.895231 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":133,"threads_started":1}
W20260926 13:44:29.898653 22542 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:29.898726 22542 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:29.898857 22543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:29.899729 22543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:29.900893 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:29.906102 22543 catalog_manager.cc:1347] Generated new cluster ID: e36ad55d8d304890b39347f1c3bb3502
I20260926 13:44:29.906181 22543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:29.912693 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:29.903302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:38203 (local address 127.0.0.1:45055)
0926 13:44:29.903418 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:29.903423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:29.903657 (+ 234us) server_negotiation.cc:407] Connection header received
0926 13:44:29.903856 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:29.903860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:29.903911 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:29.903999 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:29.904666 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:29.905554 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:29.906575 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:29.906984 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:29.907105 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:29.908832 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:29.908836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:29.908839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:29.908868 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:29.910713 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:29.911260 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:29.911264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:29.911266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:29.911366 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:29.911717 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:29.911721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:29.911723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:29.911982 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:29.912159 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:29.912407 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 13:44:29.912515 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:44:29.921212 22543 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:29.922267 22543 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:29.930248 22543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f721449ddc2645f5bfa3d196dd99d88d: Generated new TSK 0
I20260926 13:44:29.931205 22543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:29.942979 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 22455
W20260926 13:44:29.948990 22440 connection.cc:535] client connection to 127.0.0.1:45055 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45055): Connection reset by peer (error 104)
2026-09-26T13:44:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:30 dist-test-slave-1wns krb5kdc[22554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:30 dist-test-slave-1wns krb5kdc[22554](info): set up 2 sockets
Sep 26 13:44:30 dist-test-slave-1wns krb5kdc[22554](info): commencing operation
krb5kdc: starting...
W20260926 13:44:31.012398 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:31 dist-test-slave-1wns krb5kdc[22554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:44:31.144076 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:31.262745 22567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:31.263095 22567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:31.263190 22567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:31.267385 22567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:31.267504 22567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:31.267544 22567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:31.267601 22567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:31.267638 22567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:31.273592 22567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38931
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:31.274837 22567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:31.276021 22567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:31.285147 22575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:31.285151 22573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:31.285193 22572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:31.285480 22567 server_base.cc:1034] running on GCE node
I20260926 13:44:31.286194 22567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:31.287413 22567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:31.288604 22567 hybrid_clock.cc:648] HybridClock initialized: now 1790430271288569 us; error 54 us; skew 500 ppm
Sep 26 13:44:31 dist-test-slave-1wns krb5kdc[22554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:31.298604 22567 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:44:31.299950 22567 webserver.cc:460] Webserver started at http://127.0.0.1:43239/ using document root <none> and password file <none>
I20260926 13:44:31.300544 22567 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:31.300611 22567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:31.300796 22567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:31.302945 22567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "333c0e60d718480f949b765ddf1cae94"
format_stamp: "Formatted at 2026-09-26 13:44:31 on dist-test-slave-1wns"
server_key: "eeb372234deb3a0be45730398faf3c5d"
server_key_iv: "51e70721d66014587bac67fac2276f61"
server_key_version: "encryptionkey@0"
I20260926 13:44:31.303486 22567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "333c0e60d718480f949b765ddf1cae94"
format_stamp: "Formatted at 2026-09-26 13:44:31 on dist-test-slave-1wns"
server_key: "eeb372234deb3a0be45730398faf3c5d"
server_key_iv: "51e70721d66014587bac67fac2276f61"
server_key_version: "encryptionkey@0"
I20260926 13:44:31.307556 22567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 13:44:31.310369 22582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:31.311910 22567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:31.312063 22567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "333c0e60d718480f949b765ddf1cae94"
format_stamp: "Formatted at 2026-09-26 13:44:31 on dist-test-slave-1wns"
server_key: "eeb372234deb3a0be45730398faf3c5d"
server_key_iv: "51e70721d66014587bac67fac2276f61"
server_key_version: "encryptionkey@0"
I20260926 13:44:31.312196 22567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:31.331318 22567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:31.335198 22567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:31.335435 22567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:31.343861 22567 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38931
I20260926 13:44:31.344317 22634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38931 every 8 connection(s)
I20260926 13:44:31.345214 22567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:31.348444 22635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:31.351604 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 22567
I20260926 13:44:31.351769 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:31.352053 1220 external_mini_cluster.cc:1428] Setting key c499580967c11021ce7d1a13a5851677
I20260926 13:44:31.354882 22635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94: Bootstrap starting.
I20260926 13:44:31.357712 22635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:31.358618 22635 log.cc:824] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:31.361254 22635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94: No bootstrap required, opened a new log
I20260926 13:44:31.362182 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:31.353520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:34585 (local address 127.0.0.1:38931)
0926 13:44:31.353972 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:31.353989 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:31.354027 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:44:31.354681 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:31.354698 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:31.355126 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:31.355427 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:31.355793 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:31.356902 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:31.358247 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:31.358854 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:31.359017 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:31.360075 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:31.360082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:31.360100 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:31.360626 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:31.360697 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:31.360728 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:31.360867 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:31.361037 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 13:44:31.361184 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":143,"threads_started":1}
I20260926 13:44:31.365736 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:31.365744 22635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333c0e60d718480f949b765ddf1cae94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38931 } }
I20260926 13:44:31.366045 22635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:31.366123 22635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333c0e60d718480f949b765ddf1cae94, State: Initialized, Role: FOLLOWER
I20260926 13:44:31.366616 22635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333c0e60d718480f949b765ddf1cae94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38931 } }
I20260926 13:44:31.366752 22635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:31.366838 22635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:31.366959 22635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:31.368263 22635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333c0e60d718480f949b765ddf1cae94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38931 } }
I20260926 13:44:31.368666 22635 leader_election.cc:302] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 333c0e60d718480f949b765ddf1cae94; no voters:
I20260926 13:44:31.369081 22635 leader_election.cc:288] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:31.369139 22645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:31.369338 22645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [term 1 LEADER]: Becoming Leader. State: Replica: 333c0e60d718480f949b765ddf1cae94, State: Running, Role: LEADER
I20260926 13:44:31.369684 22645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333c0e60d718480f949b765ddf1cae94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38931 } }
I20260926 13:44:31.370564 22635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:31.371495 22646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 333c0e60d718480f949b765ddf1cae94. Latest consensus state: current_term: 1 leader_uuid: "333c0e60d718480f949b765ddf1cae94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333c0e60d718480f949b765ddf1cae94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38931 } } }
I20260926 13:44:31.371644 22646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:31.372090 22654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:31.372040 22647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "333c0e60d718480f949b765ddf1cae94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333c0e60d718480f949b765ddf1cae94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38931 } } }
I20260926 13:44:31.372156 22647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:31.374078 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:31.368313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:48607 (local address 127.0.0.1:38931)
0926 13:44:31.368494 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:31.368499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:31.368715 (+ 216us) server_negotiation.cc:407] Connection header received
0926 13:44:31.368893 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:31.368897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:31.368959 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:31.369055 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:31.369439 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:31.370240 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:31.371554 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:31.372027 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:31.372117 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:31.373458 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:31.373462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:31.373465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:31.373481 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:31.373515 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:31.373535 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:31.373636 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:31.373897 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 13:44:31.373939 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":86}
I20260926 13:44:31.375844 22654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:31.382309 22654 catalog_manager.cc:1347] Generated new cluster ID: 74cf65e0ced54318a524a1717bb4d29a
I20260926 13:44:31.382418 22654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:31.398188 22654 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:31.399676 22654 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:31.418224 22654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 333c0e60d718480f949b765ddf1cae94: Generated new TSK 0
I20260926 13:44:31.419224 22654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:31.423158 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 22567
W20260926 13:44:31.428678 22552 connection.cc:535] client connection to 127.0.0.1:38931 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38931): Connection reset by peer (error 104)
2026-09-26T13:44:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1479 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:31 dist-test-slave-1wns krb5kdc[22666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:31 dist-test-slave-1wns krb5kdc[22666](info): set up 2 sockets
Sep 26 13:44:31 dist-test-slave-1wns krb5kdc[22666](info): commencing operation
krb5kdc: starting...
W20260926 13:44:32.492900 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:32 dist-test-slave-1wns krb5kdc[22666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:44:32.632894 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:32.756680 22679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:32.757007 22679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:32.757054 22679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:32.761435 22679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:32.761518 22679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:32.761538 22679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:32.761554 22679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:32.761570 22679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:32.767333 22679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44215
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:32.768654 22679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:32.769773 22679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:32.778901 22684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:32.778929 22687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:32.779158 22679 server_base.cc:1034] running on GCE node
W20260926 13:44:32.778901 22685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:32.779999 22679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:32.781265 22679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:32.782471 22679 hybrid_clock.cc:648] HybridClock initialized: now 1790430272782455 us; error 29 us; skew 500 ppm
Sep 26 13:44:32 dist-test-slave-1wns krb5kdc[22666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:32.785600 22679 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:44:32.786723 22679 webserver.cc:460] Webserver started at http://127.0.0.1:37239/ using document root <none> and password file <none>
I20260926 13:44:32.787353 22679 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:32.787457 22679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:32.787716 22679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:32.790082 22679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "2e2851744a354c5089c2dd0640a49de2"
format_stamp: "Formatted at 2026-09-26 13:44:32 on dist-test-slave-1wns"
server_key: "9d8a02972da8c5566f59821c8f8b5829"
server_key_iv: "2f6efe25641fbb0db5056ee4a64ae643"
server_key_version: "encryptionkey@0"
I20260926 13:44:32.790764 22679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "2e2851744a354c5089c2dd0640a49de2"
format_stamp: "Formatted at 2026-09-26 13:44:32 on dist-test-slave-1wns"
server_key: "9d8a02972da8c5566f59821c8f8b5829"
server_key_iv: "2f6efe25641fbb0db5056ee4a64ae643"
server_key_version: "encryptionkey@0"
I20260926 13:44:32.795244 22679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 13:44:32.798195 22694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:32.799567 22679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:44:32.799741 22679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "2e2851744a354c5089c2dd0640a49de2"
format_stamp: "Formatted at 2026-09-26 13:44:32 on dist-test-slave-1wns"
server_key: "9d8a02972da8c5566f59821c8f8b5829"
server_key_iv: "2f6efe25641fbb0db5056ee4a64ae643"
server_key_version: "encryptionkey@0"
I20260926 13:44:32.799940 22679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:32.805306 22679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:32.809265 22679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:32.809499 22679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:32.818073 22679 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44215
I20260926 13:44:32.818082 22746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44215 every 8 connection(s)
I20260926 13:44:32.819548 22679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:32.821117 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 22679
I20260926 13:44:32.821269 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:32.821586 1220 external_mini_cluster.cc:1428] Setting key b7a028bd0782ef7c4573a836a5a17203
I20260926 13:44:32.823354 22747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:32.827657 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:32.823206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:55627 (local address 127.0.0.1:44215)
0926 13:44:32.823662 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:32.823680 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:32.823720 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:44:32.824479 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:32.824498 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:32.824817 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:32.825133 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:32.826065 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:32.826069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:32.826082 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:32.826094 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:32.826153 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:32.826175 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:32.826361 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:32.826421 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 13:44:32.826626 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":149,"threads_started":1}
I20260926 13:44:32.830356 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2: Bootstrap starting.
I20260926 13:44:32.831166 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:32.832930 22747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:32.833904 22747 log.cc:824] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:32.835673 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:32.833611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:37019 (local address 127.0.0.1:44215)
0926 13:44:32.833762 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:32.833767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:32.834018 (+ 251us) server_negotiation.cc:407] Connection header received
0926 13:44:32.834197 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:32.834202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:32.834339 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:32.834444 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:32.835234 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:32.835238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:32.835241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:32.835254 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:32.835284 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:32.835298 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:32.835402 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:32.835493 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 13:44:32.835542 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:44:32.836524 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2: No bootstrap required, opened a new log
I20260926 13:44:32.840759 22747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2851744a354c5089c2dd0640a49de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44215 } }
I20260926 13:44:32.841063 22747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:32.841176 22747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2851744a354c5089c2dd0640a49de2, State: Initialized, Role: FOLLOWER
I20260926 13:44:32.841719 22747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2851744a354c5089c2dd0640a49de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44215 } }
I20260926 13:44:32.841885 22747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:32.841980 22747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:32.842120 22747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:32.843266 22747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2851744a354c5089c2dd0640a49de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44215 } }
I20260926 13:44:32.843664 22747 leader_election.cc:302] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e2851744a354c5089c2dd0640a49de2; no voters:
I20260926 13:44:32.844090 22747 leader_election.cc:288] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:32.844267 22757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:32.844527 22757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [term 1 LEADER]: Becoming Leader. State: Replica: 2e2851744a354c5089c2dd0640a49de2, State: Running, Role: LEADER
I20260926 13:44:32.844842 22757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2851744a354c5089c2dd0640a49de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44215 } }
I20260926 13:44:32.845534 22747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:32.846699 22758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e2851744a354c5089c2dd0640a49de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2851744a354c5089c2dd0640a49de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44215 } } }
I20260926 13:44:32.846853 22758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:32.847151 22759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e2851744a354c5089c2dd0640a49de2. Latest consensus state: current_term: 1 leader_uuid: "2e2851744a354c5089c2dd0640a49de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2851744a354c5089c2dd0640a49de2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44215 } } }
I20260926 13:44:32.847260 22759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:32.847338 22766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:32.850730 22766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:32.856559 22766 catalog_manager.cc:1347] Generated new cluster ID: 5dd4763008a445e79b15dd40b669b1bb
I20260926 13:44:32.856653 22766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:32.868220 22766 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:32.868643 22766 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:32.875973 22766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e2851744a354c5089c2dd0640a49de2: Generated new TSK 0
I20260926 13:44:32.876919 22766 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:32.880702 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 22679
2026-09-26T13:44:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:32 dist-test-slave-1wns krb5kdc[22778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:32 dist-test-slave-1wns krb5kdc[22778](info): set up 2 sockets
Sep 26 13:44:32 dist-test-slave-1wns krb5kdc[22778](info): commencing operation
krb5kdc: starting...
W20260926 13:44:33.964217 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:34 dist-test-slave-1wns krb5kdc[22778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:44:34.099005 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:35969
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:35969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:34.217011 22791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:34.217478 22791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:34.217763 22791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:34.222615 22791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:34.222733 22791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:34.222774 22791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:34.222813 22791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:34.222851 22791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:34.228530 22791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:35969
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:35969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:34.229723 22791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:34.230799 22791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:34.240442 22796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:34.240483 22797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:34.240486 22799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:34.240849 22791 server_base.cc:1034] running on GCE node
I20260926 13:44:34.241576 22791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:34.242928 22791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:34.244153 22791 hybrid_clock.cc:648] HybridClock initialized: now 1790430274244137 us; error 31 us; skew 500 ppm
Sep 26 13:44:34 dist-test-slave-1wns krb5kdc[22778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430274, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:34.247233 22791 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:44:34.248396 22791 webserver.cc:460] Webserver started at http://127.1.49.62:39903/ using document root <none> and password file <none>
I20260926 13:44:34.249019 22791 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:34.249117 22791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:34.249375 22791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:34.251590 22791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "f02be8e613f74ca39da21a5a3c7e1e88"
format_stamp: "Formatted at 2026-09-26 13:44:34 on dist-test-slave-1wns"
server_key: "c1fe233d32f17ebc5885b0a136505c0f"
server_key_iv: "eaa94fc9c86dace64ba885d30a335352"
server_key_version: "encryptionkey@0"
I20260926 13:44:34.252250 22791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "f02be8e613f74ca39da21a5a3c7e1e88"
format_stamp: "Formatted at 2026-09-26 13:44:34 on dist-test-slave-1wns"
server_key: "c1fe233d32f17ebc5885b0a136505c0f"
server_key_iv: "eaa94fc9c86dace64ba885d30a335352"
server_key_version: "encryptionkey@0"
I20260926 13:44:34.256268 22791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:44:34.259019 22806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:34.260496 22791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:34.260651 22791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "f02be8e613f74ca39da21a5a3c7e1e88"
format_stamp: "Formatted at 2026-09-26 13:44:34 on dist-test-slave-1wns"
server_key: "c1fe233d32f17ebc5885b0a136505c0f"
server_key_iv: "eaa94fc9c86dace64ba885d30a335352"
server_key_version: "encryptionkey@0"
I20260926 13:44:34.260787 22791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:34.271036 22791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:34.274540 22791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:34.274761 22791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:34.282425 22791 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:35969
I20260926 13:44:34.282445 22858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:35969 every 8 connection(s)
I20260926 13:44:34.283464 22791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:34.286078 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 22791
I20260926 13:44:34.286190 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:34.286523 1220 external_mini_cluster.cc:1428] Setting key ebd4091718db549672af9a8b1c7a7625
I20260926 13:44:34.286525 22859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:34.294365 22859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88: Bootstrap starting.
Sep 26 13:44:34 dist-test-slave-1wns krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:34.297207 22859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:34.298182 22859 log.cc:824] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:34.300997 22859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88: No bootstrap required, opened a new log
I20260926 13:44:34.304005 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:34.288203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42523 (local address 127.1.49.62:35969)
0926 13:44:34.288705 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:34.288723 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:34.288768 (+ 45us) server_negotiation.cc:407] Connection header received
0926 13:44:34.289614 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:34.289640 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:34.290112 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:34.290616 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:34.291469 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:34.293000 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:34.293969 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:34.294667 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:34.294831 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:34.297814 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:34.297822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:34.297841 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:34.297877 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:34.300191 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:34.300825 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:34.300830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:34.300837 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:34.300979 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:34.301496 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:34.301499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:34.301501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:34.302031 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:34.302250 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:34.302590 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 13:44:34.302841 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":149,"threads_started":1}
I20260926 13:44:34.305653 22859 raft_consensus.cc:348] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02be8e613f74ca39da21a5a3c7e1e88" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35969 } }
I20260926 13:44:34.305953 22859 raft_consensus.cc:374] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:34.306015 22859 raft_consensus.cc:729] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02be8e613f74ca39da21a5a3c7e1e88, State: Initialized, Role: FOLLOWER
I20260926 13:44:34.306529 22859 consensus_queue.cc:260] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02be8e613f74ca39da21a5a3c7e1e88" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35969 } }
I20260926 13:44:34.306666 22859 raft_consensus.cc:388] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:34.306748 22859 raft_consensus.cc:482] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:34.306869 22859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:34.307843 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:34.308130 22859 raft_consensus.cc:504] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02be8e613f74ca39da21a5a3c7e1e88" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35969 } }
I20260926 13:44:34.308493 22859 leader_election.cc:302] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f02be8e613f74ca39da21a5a3c7e1e88; no voters:
I20260926 13:44:34.308876 22859 leader_election.cc:288] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:34.308952 22864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:34.309190 22864 raft_consensus.cc:686] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [term 1 LEADER]: Becoming Leader. State: Replica: f02be8e613f74ca39da21a5a3c7e1e88, State: Running, Role: LEADER
I20260926 13:44:34.309556 22864 consensus_queue.cc:237] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02be8e613f74ca39da21a5a3c7e1e88" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35969 } }
I20260926 13:44:34.310474 22859 sys_catalog.cc:564] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:34.311203 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f02be8e613f74ca39da21a5a3c7e1e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02be8e613f74ca39da21a5a3c7e1e88" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35969 } } }
I20260926 13:44:34.311409 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:34.311729 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f02be8e613f74ca39da21a5a3c7e1e88. Latest consensus state: current_term: 1 leader_uuid: "f02be8e613f74ca39da21a5a3c7e1e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02be8e613f74ca39da21a5a3c7e1e88" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35969 } } }
I20260926 13:44:34.311939 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:34.312290 22878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:34.315791 22878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:34.320013 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:34.310256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38285 (local address 127.1.49.62:35969)
0926 13:44:34.310405 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:34.310409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:34.310638 (+ 229us) server_negotiation.cc:407] Connection header received
0926 13:44:34.310833 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:34.310837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:34.310886 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:34.310940 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:34.311993 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:34.312750 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:34.313968 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:34.314229 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:34.314355 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:34.316275 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:34.316279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:34.316281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:34.316311 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:34.317873 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:34.318490 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:34.318494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:34.318496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:34.318586 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:34.318976 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:34.318979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:34.318981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:34.319208 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:34.319349 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:34.319732 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 13:44:34.319865 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 13:44:34.322314 22878 catalog_manager.cc:1347] Generated new cluster ID: 5833e71166b04badaa3c7b3f8f22d400
I20260926 13:44:34.322389 22878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:34.337631 22878 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:34.338995 22878 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:34.343325 22878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f02be8e613f74ca39da21a5a3c7e1e88: Generated new TSK 0
I20260926 13:44:34.344172 22878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:34.358465 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 22791
W20260926 13:44:34.364135 22776 connection.cc:535] client connection to 127.1.49.62:35969 recv error: Network error: failed to read from TLS socket (remote: 127.1.49.62:35969): Connection reset by peer (error 104)
2026-09-26T13:44:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1477 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:34 dist-test-slave-1wns krb5kdc[22890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:34 dist-test-slave-1wns krb5kdc[22890](info): set up 2 sockets
Sep 26 13:44:34 dist-test-slave-1wns krb5kdc[22890](info): commencing operation
krb5kdc: starting...
W20260926 13:44:35.464375 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:35 dist-test-slave-1wns krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:44:35.604034 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:41711
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:41711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:35.728040 22903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:35.728406 22903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:35.728523 22903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:35.733099 22903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:35.733224 22903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:35.733264 22903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:35.733328 22903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:35.733366 22903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:35.739079 22903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:41711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:41711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:35.740429 22903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:35.741551 22903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:44:35.751108 22903 server_base.cc:1034] running on GCE node
W20260926 13:44:35.751073 22908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:35.751359 22909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:35.751160 22911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:35.752128 22903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:35.753436 22903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:35.754627 22903 hybrid_clock.cc:648] HybridClock initialized: now 1790430275754608 us; error 38 us; skew 500 ppm
Sep 26 13:44:35 dist-test-slave-1wns krb5kdc[22890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430275, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:35.757855 22903 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:44:35.758862 22903 webserver.cc:460] Webserver started at http://127.1.49.62:44219/ using document root <none> and password file <none>
I20260926 13:44:35.759531 22903 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:35.759601 22903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:35.759886 22903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:35.762140 22903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "ae5bf42d539444ea8f67b805985b3e04"
format_stamp: "Formatted at 2026-09-26 13:44:35 on dist-test-slave-1wns"
server_key: "b0e8acfb75e7725a5af359ef22b49a3f"
server_key_iv: "fb6c4585b9561bcb0c996e337fa7d346"
server_key_version: "encryptionkey@0"
I20260926 13:44:35.762751 22903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "ae5bf42d539444ea8f67b805985b3e04"
format_stamp: "Formatted at 2026-09-26 13:44:35 on dist-test-slave-1wns"
server_key: "b0e8acfb75e7725a5af359ef22b49a3f"
server_key_iv: "fb6c4585b9561bcb0c996e337fa7d346"
server_key_version: "encryptionkey@0"
I20260926 13:44:35.767024 22903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 13:44:35.770289 22918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:35.772044 22903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:35.772292 22903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "ae5bf42d539444ea8f67b805985b3e04"
format_stamp: "Formatted at 2026-09-26 13:44:35 on dist-test-slave-1wns"
server_key: "b0e8acfb75e7725a5af359ef22b49a3f"
server_key_iv: "fb6c4585b9561bcb0c996e337fa7d346"
server_key_version: "encryptionkey@0"
I20260926 13:44:35.772466 22903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:35.786952 22903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:35.790999 22903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:35.791257 22903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:35.799364 22903 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:41711
I20260926 13:44:35.799563 22970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:41711 every 8 connection(s)
I20260926 13:44:35.800628 22903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:35.802012 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 22903
I20260926 13:44:35.802174 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:35.802487 1220 external_mini_cluster.cc:1428] Setting key 9ac286d15fcd587070d973c5089eb015
I20260926 13:44:35.804419 22971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:35.812244 22971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04: Bootstrap starting.
I20260926 13:44:35.813874 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:35.804229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38783 (local address 127.1.49.62:41711)
0926 13:44:35.804725 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:35.804747 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:35.804794 (+ 47us) server_negotiation.cc:407] Connection header received
0926 13:44:35.805700 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:35.805725 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:35.806195 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:35.806645 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:35.807090 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:35.808688 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:35.809739 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:35.810467 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:35.810668 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:35.811652 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:35.811662 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:35.811680 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:35.812255 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:35.812333 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:35.812368 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:35.812519 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:35.812724 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 13:44:35.812875 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":158,"threads_started":1}
I20260926 13:44:35.815186 22971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:35.816172 22971 log.cc:824] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:35.817651 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:35.818488 22971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04: No bootstrap required, opened a new log
I20260926 13:44:35.823107 22971 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5bf42d539444ea8f67b805985b3e04" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41711 } }
I20260926 13:44:35.823468 22971 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:35.823537 22971 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5bf42d539444ea8f67b805985b3e04, State: Initialized, Role: FOLLOWER
I20260926 13:44:35.824072 22971 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5bf42d539444ea8f67b805985b3e04" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41711 } }
I20260926 13:44:35.824205 22971 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:35.824249 22971 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:35.824347 22971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:35.825138 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:35.820165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35217 (local address 127.1.49.62:41711)
0926 13:44:35.820302 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:35.820307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:35.820481 (+ 174us) server_negotiation.cc:407] Connection header received
0926 13:44:35.820601 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:35.820604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:35.820665 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:35.820761 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:35.821027 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:35.821833 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:35.822718 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:35.823068 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:35.823186 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:35.824558 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:35.824562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:35.824565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:35.824581 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:35.824621 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:35.824634 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:35.824752 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:35.824930 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 13:44:35.824975 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:44:35.825560 22971 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5bf42d539444ea8f67b805985b3e04" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41711 } }
I20260926 13:44:35.825945 22971 leader_election.cc:302] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae5bf42d539444ea8f67b805985b3e04; no voters:
I20260926 13:44:35.826359 22971 leader_election.cc:288] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:35.826481 22981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:35.826757 22981 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [term 1 LEADER]: Becoming Leader. State: Replica: ae5bf42d539444ea8f67b805985b3e04, State: Running, Role: LEADER
I20260926 13:44:35.827160 22981 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5bf42d539444ea8f67b805985b3e04" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41711 } }
I20260926 13:44:35.827726 22971 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:35.829067 22982 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae5bf42d539444ea8f67b805985b3e04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5bf42d539444ea8f67b805985b3e04" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41711 } } }
I20260926 13:44:35.829214 22982 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:35.829631 22990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:35.829535 22983 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae5bf42d539444ea8f67b805985b3e04. Latest consensus state: current_term: 1 leader_uuid: "ae5bf42d539444ea8f67b805985b3e04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5bf42d539444ea8f67b805985b3e04" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41711 } } }
I20260926 13:44:35.832515 22983 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:35.833523 22990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:35.839636 22990 catalog_manager.cc:1347] Generated new cluster ID: 31811e3b8a574649a64ec418eaae9bd6
I20260926 13:44:35.839732 22990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:35.867101 22990 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:35.868356 22990 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:35.875087 22990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae5bf42d539444ea8f67b805985b3e04: Generated new TSK 0
I20260926 13:44:35.876051 22990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:35.888545 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 22903
W20260926 13:44:35.894083 22888 connection.cc:535] client connection to 127.1.49.62:41711 recv error: Network error: failed to read from TLS socket (remote: 127.1.49.62:41711): Connection reset by peer (error 104)
2026-09-26T13:44:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1529 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:35 dist-test-slave-1wns krb5kdc[23002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:35 dist-test-slave-1wns krb5kdc[23002](info): set up 2 sockets
Sep 26 13:44:35 dist-test-slave-1wns krb5kdc[23002](info): commencing operation
krb5kdc: starting...
W20260926 13:44:37.004290 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:37 dist-test-slave-1wns krb5kdc[23002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:44:37.140010 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:44267
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:44267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:37.255988 23015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:37.256356 23015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:37.256438 23015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:37.260696 23015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:37.260824 23015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:37.260864 23015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:37.260905 23015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:37.260941 23015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:37.266515 23015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:44267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:44267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:37.267740 23015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:37.268909 23015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:37.278769 23023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:37.278769 23021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:37.278766 23020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:37.279488 23015 server_base.cc:1034] running on GCE node
I20260926 13:44:37.280239 23015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:37.281445 23015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:37.282626 23015 hybrid_clock.cc:648] HybridClock initialized: now 1790430277282594 us; error 59 us; skew 500 ppm
Sep 26 13:44:37 dist-test-slave-1wns krb5kdc[23002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430277, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:37.290861 23015 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:44:37.292001 23015 webserver.cc:460] Webserver started at http://127.1.49.62:45695/ using document root <none> and password file <none>
I20260926 13:44:37.292683 23015 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:37.292793 23015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:37.293113 23015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:37.296486 23015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "8d9df254ffc54027a9b85ac6f155fc87"
format_stamp: "Formatted at 2026-09-26 13:44:37 on dist-test-slave-1wns"
server_key: "54a25305e6295e3723c69d55750240f3"
server_key_iv: "c85ad30140fb14d48707e97be27b574c"
server_key_version: "encryptionkey@0"
I20260926 13:44:37.297350 23015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "8d9df254ffc54027a9b85ac6f155fc87"
format_stamp: "Formatted at 2026-09-26 13:44:37 on dist-test-slave-1wns"
server_key: "54a25305e6295e3723c69d55750240f3"
server_key_iv: "c85ad30140fb14d48707e97be27b574c"
server_key_version: "encryptionkey@0"
I20260926 13:44:37.302386 23015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 13:44:37.305240 23030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:37.306793 23015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:44:37.307006 23015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "8d9df254ffc54027a9b85ac6f155fc87"
format_stamp: "Formatted at 2026-09-26 13:44:37 on dist-test-slave-1wns"
server_key: "54a25305e6295e3723c69d55750240f3"
server_key_iv: "c85ad30140fb14d48707e97be27b574c"
server_key_version: "encryptionkey@0"
I20260926 13:44:37.307147 23015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:37.312507 23015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:37.316220 23015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:37.316452 23015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:37.324512 23015 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:44267
I20260926 13:44:37.324538 23082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:44267 every 8 connection(s)
I20260926 13:44:37.325630 23015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:37.327972 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 23015
I20260926 13:44:37.328117 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:37.328614 1220 external_mini_cluster.cc:1428] Setting key 7e88792fcc03741d09ecb77f5f286ad9
I20260926 13:44:37.329147 23083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:37.334452 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:37.330131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54519 (local address 127.1.49.62:44267)
0926 13:44:37.330665 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:37.330682 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:37.330721 (+ 39us) server_negotiation.cc:407] Connection header received
0926 13:44:37.331477 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:37.331497 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:37.331895 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:37.332290 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:37.333155 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:37.333160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:37.333173 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:37.333185 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:37.333243 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:37.333264 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:37.333438 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:37.333514 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 13:44:37.333685 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":171,"threads_started":1}
I20260926 13:44:37.336447 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87: Bootstrap starting.
I20260926 13:44:37.337709 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:37.339421 23083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:37.340428 23083 log.cc:824] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:37.342119 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:37.340244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49399 (local address 127.1.49.62:44267)
0926 13:44:37.340404 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:37.340409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:37.340666 (+ 257us) server_negotiation.cc:407] Connection header received
0926 13:44:37.340810 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:37.340812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:37.340876 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:37.340970 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:37.341734 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:37.341738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:37.341740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:37.341752 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:37.341780 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:37.341793 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:37.341885 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:37.341945 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 13:44:37.341983 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 13:44:37.342890 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87: No bootstrap required, opened a new log
I20260926 13:44:37.346801 23083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9df254ffc54027a9b85ac6f155fc87" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44267 } }
I20260926 13:44:37.347060 23083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:37.347097 23083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d9df254ffc54027a9b85ac6f155fc87, State: Initialized, Role: FOLLOWER
I20260926 13:44:37.347479 23083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9df254ffc54027a9b85ac6f155fc87" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44267 } }
I20260926 13:44:37.347633 23083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:37.347704 23083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:37.347844 23083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:37.349102 23083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9df254ffc54027a9b85ac6f155fc87" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44267 } }
I20260926 13:44:37.349438 23083 leader_election.cc:302] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d9df254ffc54027a9b85ac6f155fc87; no voters:
I20260926 13:44:37.349718 23083 leader_election.cc:288] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:37.349879 23093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:37.350138 23093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [term 1 LEADER]: Becoming Leader. State: Replica: 8d9df254ffc54027a9b85ac6f155fc87, State: Running, Role: LEADER
I20260926 13:44:37.350464 23093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9df254ffc54027a9b85ac6f155fc87" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44267 } }
I20260926 13:44:37.351043 23083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:37.351750 23093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d9df254ffc54027a9b85ac6f155fc87. Latest consensus state: current_term: 1 leader_uuid: "8d9df254ffc54027a9b85ac6f155fc87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9df254ffc54027a9b85ac6f155fc87" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44267 } } }
I20260926 13:44:37.351969 23093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:37.352308 23094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d9df254ffc54027a9b85ac6f155fc87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9df254ffc54027a9b85ac6f155fc87" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 44267 } } }
I20260926 13:44:37.352392 23102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:37.352509 23094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:37.356009 23102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:37.363085 23102 catalog_manager.cc:1347] Generated new cluster ID: 870f83f8ec51437a90fedb554b7220ed
I20260926 13:44:37.363198 23102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:37.383040 23102 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:37.383502 23102 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:37.389349 23102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d9df254ffc54027a9b85ac6f155fc87: Generated new TSK 0
I20260926 13:44:37.390386 23102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:37.405691 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 23015
2026-09-26T13:44:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1518 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:37 dist-test-slave-1wns krb5kdc[23114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:37 dist-test-slave-1wns krb5kdc[23114](info): set up 2 sockets
Sep 26 13:44:37 dist-test-slave-1wns krb5kdc[23114](info): commencing operation
krb5kdc: starting...
W20260926 13:44:38.484361 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:38 dist-test-slave-1wns krb5kdc[23114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:44:38.621954 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:35287
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:35287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:38.744522 23127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:38.744887 23127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:38.744935 23127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:38.749490 23127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:38.749579 23127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:38.749600 23127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:38.749619 23127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:38.749634 23127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:38.755484 23127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:35287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:35287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:38.756726 23127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:38.757757 23127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:38.767148 23133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:38.767144 23135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:38.767153 23132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:38.767740 23127 server_base.cc:1034] running on GCE node
I20260926 13:44:38.768435 23127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:38.769711 23127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:38.770910 23127 hybrid_clock.cc:648] HybridClock initialized: now 1790430278770871 us; error 48 us; skew 500 ppm
Sep 26 13:44:38 dist-test-slave-1wns krb5kdc[23114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430278, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:38.774292 23127 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:44:38.775393 23127 webserver.cc:460] Webserver started at http://127.1.49.62:43935/ using document root <none> and password file <none>
I20260926 13:44:38.776115 23127 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:38.776188 23127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:38.776435 23127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:38.778648 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "2ba5a20115ec4219a970117bd8f88b15"
format_stamp: "Formatted at 2026-09-26 13:44:38 on dist-test-slave-1wns"
server_key: "e0d607d0483fc59c8e91297a42e5f284"
server_key_iv: "32dfc5f08afb8e390635f320cd132ea6"
server_key_version: "encryptionkey@0"
I20260926 13:44:38.779240 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "2ba5a20115ec4219a970117bd8f88b15"
format_stamp: "Formatted at 2026-09-26 13:44:38 on dist-test-slave-1wns"
server_key: "e0d607d0483fc59c8e91297a42e5f284"
server_key_iv: "32dfc5f08afb8e390635f320cd132ea6"
server_key_version: "encryptionkey@0"
I20260926 13:44:38.783299 23127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 13:44:38.786147 23142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:38.787611 23127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:44:38.787726 23127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "2ba5a20115ec4219a970117bd8f88b15"
format_stamp: "Formatted at 2026-09-26 13:44:38 on dist-test-slave-1wns"
server_key: "e0d607d0483fc59c8e91297a42e5f284"
server_key_iv: "32dfc5f08afb8e390635f320cd132ea6"
server_key_version: "encryptionkey@0"
I20260926 13:44:38.787933 23127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:38.805073 23127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:38.809382 23127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:38.809636 23127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:38.818151 23127 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:35287
I20260926 13:44:38.818217 23194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:35287 every 8 connection(s)
I20260926 13:44:38.819324 23127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:38.820043 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 23127
I20260926 13:44:38.820221 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:38.820572 1220 external_mini_cluster.cc:1428] Setting key cafc2dfa6215efb6a4bb035068cfd8ae
I20260926 13:44:38.822759 23195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:38.829926 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15: Bootstrap starting.
Sep 26 13:44:38 dist-test-slave-1wns krb5kdc[23114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:38.832821 23195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:38.833811 23195 log.cc:824] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:38.836659 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15: No bootstrap required, opened a new log
I20260926 13:44:38.837419 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:38.822112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39543 (local address 127.1.49.62:35287)
0926 13:44:38.822568 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:38.822597 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:38.822635 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:44:38.823398 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:38.823419 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:38.823871 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:38.824225 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:38.825212 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:38.826461 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:38.827692 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:38.828303 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:38.828437 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:38.831493 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:38.831500 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:38.831518 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:38.831557 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:38.833816 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:38.834454 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:38.834459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:38.834464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:38.834579 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:38.834959 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:38.834966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:38.834971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:38.835491 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:38.835689 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:38.836072 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 13:44:38.836400 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":166,"threads_started":1}
I20260926 13:44:38.840812 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:38.841246 23195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba5a20115ec4219a970117bd8f88b15" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35287 } }
I20260926 13:44:38.841507 23195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:38.841593 23195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba5a20115ec4219a970117bd8f88b15, State: Initialized, Role: FOLLOWER
I20260926 13:44:38.842088 23195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba5a20115ec4219a970117bd8f88b15" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35287 } }
I20260926 13:44:38.842257 23195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:38.842361 23195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:38.842490 23195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:38.843768 23195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba5a20115ec4219a970117bd8f88b15" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35287 } }
I20260926 13:44:38.844197 23195 leader_election.cc:302] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ba5a20115ec4219a970117bd8f88b15; no voters:
I20260926 13:44:38.844586 23195 leader_election.cc:288] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:38.844667 23205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:38.845820 23205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [term 1 LEADER]: Becoming Leader. State: Replica: 2ba5a20115ec4219a970117bd8f88b15, State: Running, Role: LEADER
I20260926 13:44:38.846072 23195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:38.846165 23205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba5a20115ec4219a970117bd8f88b15" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35287 } }
I20260926 13:44:38.848387 23206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ba5a20115ec4219a970117bd8f88b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba5a20115ec4219a970117bd8f88b15" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35287 } } }
I20260926 13:44:38.849767 23206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:38.850193 23207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ba5a20115ec4219a970117bd8f88b15. Latest consensus state: current_term: 1 leader_uuid: "2ba5a20115ec4219a970117bd8f88b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba5a20115ec4219a970117bd8f88b15" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35287 } } }
I20260926 13:44:38.850338 23207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15 [sys.catalog]: This master's current role is: LEADER
W20260926 13:44:38.852370 23219 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:38.852473 23219 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:38.852560 23221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:38.853466 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:38.843277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58111 (local address 127.1.49.62:35287)
0926 13:44:38.843407 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:38.843412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:38.843583 (+ 171us) server_negotiation.cc:407] Connection header received
0926 13:44:38.843681 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:38.843687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:38.843749 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:38.843885 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:38.844802 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:38.845560 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:38.846775 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:38.847235 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:38.847400 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:38.849779 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:38.849783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:38.849785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:38.849813 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:38.851507 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:38.852076 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:38.852080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:38.852082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:38.852171 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:38.852488 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:38.852493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:38.852498 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:38.852766 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:38.852914 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:38.853165 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 13:44:38.853289 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":45}
I20260926 13:44:38.853797 23221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:38.860333 23221 catalog_manager.cc:1347] Generated new cluster ID: bac68377a80343c99f53ba09e0c7e1f6
I20260926 13:44:38.860447 23221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:38.875838 23221 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:38.876991 23221 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:38.882694 23221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ba5a20115ec4219a970117bd8f88b15: Generated new TSK 0
I20260926 13:44:38.883488 23221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:38.890900 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 23127
2026-09-26T13:44:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:38 dist-test-slave-1wns krb5kdc[23226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:38 dist-test-slave-1wns krb5kdc[23226](info): set up 2 sockets
Sep 26 13:44:38 dist-test-slave-1wns krb5kdc[23226](info): commencing operation
krb5kdc: starting...
W20260926 13:44:39.972347 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:40 dist-test-slave-1wns krb5kdc[23226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:44:40.107424 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:35239
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:35239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:40.225533 23239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:40.225860 23239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:40.225909 23239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:40.230659 23239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:40.230746 23239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:40.230768 23239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:40.230787 23239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:40.230805 23239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:40.236886 23239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:35239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:35239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:40.238082 23239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:40.239177 23239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:40.248361 23247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:40.248404 23245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:40.248420 23244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:40.248533 23239 server_base.cc:1034] running on GCE node
I20260926 13:44:40.249257 23239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:40.250502 23239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:40.251709 23239 hybrid_clock.cc:648] HybridClock initialized: now 1790430280251675 us; error 47 us; skew 500 ppm
Sep 26 13:44:40 dist-test-slave-1wns krb5kdc[23226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430280, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:40.254681 23239 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:44:40.255565 23239 webserver.cc:460] Webserver started at http://127.1.49.62:39523/ using document root <none> and password file <none>
I20260926 13:44:40.256170 23239 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:40.256237 23239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:40.256410 23239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:40.258445 23239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "3e88d4b436b34b6fa827dbef397af878"
format_stamp: "Formatted at 2026-09-26 13:44:40 on dist-test-slave-1wns"
server_key: "4749603d42a2dadaf459487d170d6bd4"
server_key_iv: "c42882a4ee768eb6ae54cea20263a980"
server_key_version: "encryptionkey@0"
I20260926 13:44:40.259012 23239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "3e88d4b436b34b6fa827dbef397af878"
format_stamp: "Formatted at 2026-09-26 13:44:40 on dist-test-slave-1wns"
server_key: "4749603d42a2dadaf459487d170d6bd4"
server_key_iv: "c42882a4ee768eb6ae54cea20263a980"
server_key_version: "encryptionkey@0"
I20260926 13:44:40.262996 23239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 13:44:40.265672 23254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:40.267097 23239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:44:40.267208 23239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "3e88d4b436b34b6fa827dbef397af878"
format_stamp: "Formatted at 2026-09-26 13:44:40 on dist-test-slave-1wns"
server_key: "4749603d42a2dadaf459487d170d6bd4"
server_key_iv: "c42882a4ee768eb6ae54cea20263a980"
server_key_version: "encryptionkey@0"
I20260926 13:44:40.267313 23239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:40.283957 23239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:40.287515 23239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:40.287686 23239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:40.295162 23239 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:35239
I20260926 13:44:40.295185 23306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:35239 every 8 connection(s)
I20260926 13:44:40.296185 23239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:40.299084 23307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:40.304916 23307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878: Bootstrap starting.
I20260926 13:44:40.304997 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 23239
I20260926 13:44:40.305119 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:40.305429 1220 external_mini_cluster.cc:1428] Setting key 6d634a176888f0f0de7362573d2741fe
I20260926 13:44:40.307756 23307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:40.308565 23307 log.cc:824] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:40.311349 23307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878: No bootstrap required, opened a new log
I20260926 13:44:40.315940 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:40.306933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48919 (local address 127.1.49.62:35239)
0926 13:44:40.307418 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:40.307435 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:40.307473 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:44:40.308343 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:40.308366 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:40.308813 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:40.309224 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:40.309761 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:40.310929 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:40.312111 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:40.312657 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:40.312873 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:40.313908 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:40.313919 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:40.313935 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:40.314425 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:40.314485 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:40.314511 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:40.314634 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:40.314771 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 13:44:40.315023 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":144,"threads_started":1}
I20260926 13:44:40.316313 23307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e88d4b436b34b6fa827dbef397af878" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35239 } }
I20260926 13:44:40.316624 23307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:40.316717 23307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e88d4b436b34b6fa827dbef397af878, State: Initialized, Role: FOLLOWER
I20260926 13:44:40.317284 23307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e88d4b436b34b6fa827dbef397af878" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35239 } }
I20260926 13:44:40.317487 23307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:40.317552 23307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:40.317655 23307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:40.319012 23307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e88d4b436b34b6fa827dbef397af878" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35239 } }
I20260926 13:44:40.319446 23307 leader_election.cc:302] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e88d4b436b34b6fa827dbef397af878; no voters:
I20260926 13:44:40.319609 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:40.319909 23307 leader_election.cc:288] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:40.319990 23312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:40.320230 23312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [term 1 LEADER]: Becoming Leader. State: Replica: 3e88d4b436b34b6fa827dbef397af878, State: Running, Role: LEADER
I20260926 13:44:40.320677 23312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e88d4b436b34b6fa827dbef397af878" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35239 } }
I20260926 13:44:40.321460 23307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:40.322502 23314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e88d4b436b34b6fa827dbef397af878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e88d4b436b34b6fa827dbef397af878" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35239 } } }
I20260926 13:44:40.322655 23314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:40.323063 23313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e88d4b436b34b6fa827dbef397af878. Latest consensus state: current_term: 1 leader_uuid: "3e88d4b436b34b6fa827dbef397af878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e88d4b436b34b6fa827dbef397af878" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 35239 } } }
I20260926 13:44:40.323123 23326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:40.323168 23313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:40.326884 23326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:40.327865 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:40.322205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45585 (local address 127.1.49.62:35239)
0926 13:44:40.322345 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:40.322352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:40.322640 (+ 288us) server_negotiation.cc:407] Connection header received
0926 13:44:40.322885 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:40.322890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:40.322948 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:40.323042 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:40.323546 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:40.324437 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:40.325534 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:40.325933 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:40.326048 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:40.327120 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:40.327124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:40.327128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:40.327144 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:40.327177 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:40.327195 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:40.327263 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:40.327626 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 13:44:40.327688 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:44:40.333518 23326 catalog_manager.cc:1347] Generated new cluster ID: 6280f5927a5347f4956f5e4edec198f0
I20260926 13:44:40.333680 23326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:40.376344 23326 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:40.377419 23326 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:40.381991 23326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e88d4b436b34b6fa827dbef397af878: Generated new TSK 0
I20260926 13:44:40.382746 23326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:40.394523 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 23239
2026-09-26T13:44:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1503 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:40 dist-test-slave-1wns krb5kdc[23338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:40 dist-test-slave-1wns krb5kdc[23338](info): set up 2 sockets
Sep 26 13:44:40 dist-test-slave-1wns krb5kdc[23338](info): commencing operation
krb5kdc: starting...
W20260926 13:44:41.476300 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:41 dist-test-slave-1wns krb5kdc[23338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:44:41.612098 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:42189
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:42189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:41.729918 23351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:41.730299 23351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:41.730394 23351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:41.734808 23351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:41.734943 23351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:41.734983 23351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:41.735024 23351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:41.735056 23351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:41.740955 23351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:42189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:42189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:41.742262 23351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:41.743383 23351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:41.752910 23356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:41.752842 23359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:41.752893 23357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:41.752964 23351 server_base.cc:1034] running on GCE node
I20260926 13:44:41.753651 23351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:41.754915 23351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:41.756117 23351 hybrid_clock.cc:648] HybridClock initialized: now 1790430281756098 us; error 38 us; skew 500 ppm
Sep 26 13:44:41 dist-test-slave-1wns krb5kdc[23338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430281, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:41.759307 23351 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:44:41.760505 23351 webserver.cc:460] Webserver started at http://127.1.49.62:39163/ using document root <none> and password file <none>
I20260926 13:44:41.761087 23351 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:41.761159 23351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:41.761350 23351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:41.763692 23351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "cb7585ae98c14feaa9184c1358a9416f"
format_stamp: "Formatted at 2026-09-26 13:44:41 on dist-test-slave-1wns"
server_key: "8411ed7b97fd8f324e6a33a75d9a173e"
server_key_iv: "bbc0e3a5561fb60b6f916b1e71fb5700"
server_key_version: "encryptionkey@0"
I20260926 13:44:41.764408 23351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "cb7585ae98c14feaa9184c1358a9416f"
format_stamp: "Formatted at 2026-09-26 13:44:41 on dist-test-slave-1wns"
server_key: "8411ed7b97fd8f324e6a33a75d9a173e"
server_key_iv: "bbc0e3a5561fb60b6f916b1e71fb5700"
server_key_version: "encryptionkey@0"
I20260926 13:44:41.768906 23351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:44:41.771768 23366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:41.773500 23351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 13:44:41.773705 23351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "cb7585ae98c14feaa9184c1358a9416f"
format_stamp: "Formatted at 2026-09-26 13:44:41 on dist-test-slave-1wns"
server_key: "8411ed7b97fd8f324e6a33a75d9a173e"
server_key_iv: "bbc0e3a5561fb60b6f916b1e71fb5700"
server_key_version: "encryptionkey@0"
I20260926 13:44:41.773859 23351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:41.779490 23351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:41.783273 23351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:41.783582 23351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:41.791298 23351 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:42189
I20260926 13:44:41.791326 23418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:42189 every 8 connection(s)
I20260926 13:44:41.792423 23351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:41.795423 23419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:41.799494 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 23351
I20260926 13:44:41.799654 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:41.800021 1220 external_mini_cluster.cc:1428] Setting key ae3bc751bdd7a5186440198d77b03d14
I20260926 13:44:41.802157 23419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f: Bootstrap starting.
I20260926 13:44:41.804898 23419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:41.805934 23419 log.cc:824] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:41.806594 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:41.801659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59035 (local address 127.1.49.62:42189)
0926 13:44:41.802157 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:41.802172 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:41.802211 (+ 39us) server_negotiation.cc:407] Connection header received
0926 13:44:41.802990 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:41.803009 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:41.803333 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:41.803708 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:41.804989 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:41.804997 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:41.805049 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:41.805066 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:41.805165 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:41.805198 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:41.805312 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:41.805444 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 13:44:41.805628 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":161,"threads_started":1}
I20260926 13:44:41.808281 23419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f: No bootstrap required, opened a new log
I20260926 13:44:41.809799 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:41.813412 23419 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7585ae98c14feaa9184c1358a9416f" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42189 } }
I20260926 13:44:41.813728 23419 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:41.813803 23419 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb7585ae98c14feaa9184c1358a9416f, State: Initialized, Role: FOLLOWER
I20260926 13:44:41.814334 23419 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7585ae98c14feaa9184c1358a9416f" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42189 } }
I20260926 13:44:41.814548 23419 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:41.814574 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:41.812363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58459 (local address 127.1.49.62:42189)
0926 13:44:41.812562 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:41.812569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:41.812779 (+ 210us) server_negotiation.cc:407] Connection header received
0926 13:44:41.812886 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:41.812888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:41.812923 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:41.812998 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:41.814006 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:41.814009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:41.814011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:44:41.814022 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:41.814072 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:44:41.814088 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:41.814188 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:41.814352 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 13:44:41.814420 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 13:44:41.814641 23419 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:41.814762 23419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:41.815892 23419 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7585ae98c14feaa9184c1358a9416f" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42189 } }
I20260926 13:44:41.816278 23419 leader_election.cc:302] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb7585ae98c14feaa9184c1358a9416f; no voters:
I20260926 13:44:41.816645 23419 leader_election.cc:288] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:41.816857 23429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:41.817149 23429 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [term 1 LEADER]: Becoming Leader. State: Replica: cb7585ae98c14feaa9184c1358a9416f, State: Running, Role: LEADER
I20260926 13:44:41.817512 23429 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7585ae98c14feaa9184c1358a9416f" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42189 } }
I20260926 13:44:41.818027 23419 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:41.819411 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb7585ae98c14feaa9184c1358a9416f. Latest consensus state: current_term: 1 leader_uuid: "cb7585ae98c14feaa9184c1358a9416f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7585ae98c14feaa9184c1358a9416f" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42189 } } }
I20260926 13:44:41.819548 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:41.819911 23430 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb7585ae98c14feaa9184c1358a9416f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7585ae98c14feaa9184c1358a9416f" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 42189 } } }
I20260926 13:44:41.820020 23430 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f [sys.catalog]: This master's current role is: LEADER
W20260926 13:44:41.823325 23444 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:41.823419 23444 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:41.823529 23439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:41.824455 23439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:41.831230 23439 catalog_manager.cc:1347] Generated new cluster ID: 4250c7cd06a74b48bbc98e3b6a820b82
I20260926 13:44:41.831420 23439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:41.855283 23439 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:41.855690 23439 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:41.866360 23439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb7585ae98c14feaa9184c1358a9416f: Generated new TSK 0
I20260926 13:44:41.867226 23439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:44:41.878274 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 23351
2026-09-26T13:44:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1484 ms)
[----------] 18 tests from AuthTokenIssuingTest (26835 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:41 dist-test-slave-1wns krb5kdc[23450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:41 dist-test-slave-1wns krb5kdc[23450](info): set up 2 sockets
Sep 26 13:44:41 dist-test-slave-1wns krb5kdc[23450](info): commencing operation
krb5kdc: starting...
W20260926 13:44:42.976413 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:44:43.127995 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:41479
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:41479,127.1.49.61:43657,127.1.49.60:37869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:43.248987 23463 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:44:43.249351 23463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:43.249380 23463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:43.249425 23463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:43.254154 23463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:43.254243 23463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:43.254272 23463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:43.254292 23463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:43.254309 23463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:43.260411 23463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:41479,127.1.49.61:43657,127.1.49.60:37869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:41479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:43.261600 23463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:43.262712 23463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:43.272527 23471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:43.272591 23463 server_base.cc:1034] running on GCE node
W20260926 13:44:43.272560 23468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:43.272527 23469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:43.273463 23463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:43.274765 23463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:43.275914 23463 hybrid_clock.cc:648] HybridClock initialized: now 1790430283275896 us; error 34 us; skew 500 ppm
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:43.285948 23463 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:44:43.287127 23463 webserver.cc:460] Webserver started at http://127.1.49.62:32875/ using document root <none> and password file <none>
I20260926 13:44:43.287928 23463 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:43.288062 23463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:43.288368 23463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:43.290841 23463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "a41209f6d7014da29ad4999d6dcd9f33"
format_stamp: "Formatted at 2026-09-26 13:44:43 on dist-test-slave-1wns"
server_key: "6ecfeaac26f924092f29c925256512e3"
server_key_iv: "c5be4ff8de3ca24482956081fb649593"
server_key_version: "encryptionkey@0"
I20260926 13:44:43.291543 23463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "a41209f6d7014da29ad4999d6dcd9f33"
format_stamp: "Formatted at 2026-09-26 13:44:43 on dist-test-slave-1wns"
server_key: "6ecfeaac26f924092f29c925256512e3"
server_key_iv: "c5be4ff8de3ca24482956081fb649593"
server_key_version: "encryptionkey@0"
I20260926 13:44:43.297981 23463 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 13:44:43.301450 23478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:43.302992 23463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:44:43.303171 23463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "a41209f6d7014da29ad4999d6dcd9f33"
format_stamp: "Formatted at 2026-09-26 13:44:43 on dist-test-slave-1wns"
server_key: "6ecfeaac26f924092f29c925256512e3"
server_key_iv: "c5be4ff8de3ca24482956081fb649593"
server_key_version: "encryptionkey@0"
I20260926 13:44:43.303329 23463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:43.327049 23463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:43.330824 23463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:43.331099 23463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:43.339083 23463 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:41479
I20260926 13:44:43.339447 23530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:41479 every 8 connection(s)
I20260926 13:44:43.340194 23463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:43.343305 23531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:43.346095 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 23463
I20260926 13:44:43.346243 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:43.346560 1220 external_mini_cluster.cc:1428] Setting key 44e5c0860cd30e230503e30f0f4f38c9
I20260926 13:44:43.348774 23531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.49.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.61@KRBTEST.COM" created.
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:43.368297 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.352524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34680 (local address 127.1.49.62:41479)
0926 13:44:43.352947 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.352962 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.353025 (+ 63us) server_negotiation.cc:407] Connection header received
0926 13:44:43.353589 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.353599 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.353900 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.354144 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.355082 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.356671 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.358044 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.358469 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.358602 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.361915 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.361921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.361964 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.362002 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.364580 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.365178 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.365188 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.365193 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.365383 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.365725 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.365729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.365731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.366412 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.366589 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.367032 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.367353 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":271,"thread_start_us":109,"threads_started":1}
I20260926 13:44:43.368295 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.351436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41479 (local address 127.0.0.1:34680)
0926 13:44:43.352028 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.352089 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:43.353200 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:43.354159 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:43.354164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:43.354621 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:43.354902 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.354916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.356809 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.356813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.357884 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.357897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.358636 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.358639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.358858 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.359622 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:43.359640 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:43.361668 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:43.364735 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.364743 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.364754 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.365037 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.365504 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.365506 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.365507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.365622 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.366581 (+ 959us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:43.366587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:43.366728 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 13:44:43.366912 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 13:44:43.367226 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":50,"thread_start_us":155,"threads_started":1}
I20260926 13:44:43.371639 23531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:44:43.372714 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.372376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:43657 (local address 127.0.0.1:43912)
0926 13:44:43.372522 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.372605 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 13:44:43.373026 23479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.49.61:43657: connect: Connection refused (error 111)
I20260926 13:44:43.374224 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.373907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:43657 (local address 127.0.0.1:43914)
0926 13:44:43.374048 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.374131 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 13:44:43.374428 23531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.61:43657: Network error: Client connection negotiation failed: client connection to 127.1.49.61:43657: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.61.keytab.
Entry for principal kudu/127.1.49.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.61.keytab.
I20260926 13:44:43.380322 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.61:43657
--webserver_interface=127.1.49.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.61.keytab
--principal=kudu/127.1.49.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:41479,127.1.49.61:43657,127.1.49.60:37869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:44:43.423784 23531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } attempt: 1
I20260926 13:44:43.424919 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.424544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:43657 (local address 127.0.0.1:43924)
0926 13:44:43.424724 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.424805 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 13:44:43.425746 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.425519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:43657 (local address 127.0.0.1:43930)
0926 13:44:43.425643 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.425690 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 13:44:43.425956 23531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.61:43657: Network error: Client connection negotiation failed: client connection to 127.1.49.61:43657: connect: Connection refused (error 111)
I20260926 13:44:43.494158 23531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } attempt: 2
I20260926 13:44:43.495330 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.494837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:43657 (local address 127.0.0.1:43936)
0926 13:44:43.495060 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.495139 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 13:44:43.496436 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.496092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:43657 (local address 127.0.0.1:43942)
0926 13:44:43.496236 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.496363 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 13:44:43.496665 23531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.61:43657: Network error: Client connection negotiation failed: client connection to 127.1.49.61:43657: connect: Connection refused (error 111)
W20260926 13:44:43.509948 23538 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:44:43.510309 23538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:43.510370 23538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:43.510474 23538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:43.515497 23538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:43.515589 23538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:43.515611 23538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:43.515630 23538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:43.515648 23538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:43.522512 23538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:41479,127.1.49.61:43657,127.1.49.60:37869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.61.keytab
--principal=kudu/127.1.49.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.61:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.49.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:43.523859 23538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:43.525046 23538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:44:43.535593 23538 server_base.cc:1034] running on GCE node
W20260926 13:44:43.535544 23544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:43.535544 23543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:43.535553 23546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:43.536602 23538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:43.537948 23538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:43.539165 23538 hybrid_clock.cc:648] HybridClock initialized: now 1790430283539139 us; error 44 us; skew 500 ppm
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:43.542887 23538 init.cc:377] Logged in from keytab as kudu/127.1.49.61@KRBTEST.COM (short username kudu)
I20260926 13:44:43.544234 23538 webserver.cc:460] Webserver started at http://127.1.49.61:38559/ using document root <none> and password file <none>
I20260926 13:44:43.544984 23538 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:43.545091 23538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:43.545368 23538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:43.548028 23538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/data/instance:
uuid: "6ada876df0ce4230acb177d205529ac6"
format_stamp: "Formatted at 2026-09-26 13:44:43 on dist-test-slave-1wns"
server_key: "b9e45074e010a7fda1aeb25b0af83e7b"
server_key_iv: "8e32cee3147aa2f95f1338bf3a41b242"
server_key_version: "encryptionkey@0"
I20260926 13:44:43.548708 23538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/wal/instance:
uuid: "6ada876df0ce4230acb177d205529ac6"
format_stamp: "Formatted at 2026-09-26 13:44:43 on dist-test-slave-1wns"
server_key: "b9e45074e010a7fda1aeb25b0af83e7b"
server_key_iv: "8e32cee3147aa2f95f1338bf3a41b242"
server_key_version: "encryptionkey@0"
I20260926 13:44:43.553148 23538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 13:44:43.556211 23553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:43.557821 23538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 13:44:43.557999 23538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/wal
uuid: "6ada876df0ce4230acb177d205529ac6"
format_stamp: "Formatted at 2026-09-26 13:44:43 on dist-test-slave-1wns"
server_key: "b9e45074e010a7fda1aeb25b0af83e7b"
server_key_iv: "8e32cee3147aa2f95f1338bf3a41b242"
server_key_version: "encryptionkey@0"
I20260926 13:44:43.558158 23538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:43.587895 23531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } attempt: 3
I20260926 13:44:43.588924 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.588552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:43657 (local address 127.0.0.1:43958)
0926 13:44:43.588762 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.588835 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 13:44:43.589811 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.589573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:43657 (local address 127.0.0.1:43970)
0926 13:44:43.589691 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.589746 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 13:44:43.589951 23531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.61:43657: Network error: Client connection negotiation failed: client connection to 127.1.49.61:43657: connect: Connection refused (error 111)
I20260926 13:44:43.603474 23538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:43.607754 23538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:43.608057 23538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:43.616678 23538 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.61:43657
I20260926 13:44:43.617035 23605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.61:43657 every 8 connection(s)
I20260926 13:44:43.617848 23538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/data/info.pb
I20260926 13:44:43.618650 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 23538
I20260926 13:44:43.618813 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-1/wal/instance
I20260926 13:44:43.619128 1220 external_mini_cluster.cc:1428] Setting key 93ce7a5eca3a8dd78b84987120d21451
I20260926 13:44:43.621423 23606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:43.625844 23606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.49.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.60@KRBTEST.COM" created.
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.61@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:43.644845 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.628068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.1.49.62:41479)
0926 13:44:43.628242 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.628247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.629339 (+ 1092us) server_negotiation.cc:407] Connection header received
0926 13:44:43.630632 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.630640 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.630743 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.630869 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.632861 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.633744 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.635968 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.636213 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.636353 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.640311 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.640315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.640317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.640350 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.642172 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.642845 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.642851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.642857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.642943 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.643358 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.643361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.643363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.643596 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.643717 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.644485 (+ 768us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.644636 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 13:44:43.645889 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.628572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41479 (local address 127.0.0.1:34682)
0926 13:44:43.629146 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.629199 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:43.630312 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:43.631103 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:43.631116 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:43.631729 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:43.632626 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.632646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.633930 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.633937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.635766 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.635785 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.636414 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.636419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.636727 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.637557 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:43.637589 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:43.640048 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:43.642325 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.642341 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.642362 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.642705 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.643060 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.643064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.643066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.643238 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.643718 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:43.643726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:43.644119 (+ 393us) client_negotiation.cc:769] Sending connection context
0926 13:44:43.644424 (+ 305us) client_negotiation.cc:240] Negotiation successful
0926 13:44:43.644866 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":139,"threads_started":1}
I20260926 13:44:43.647710 23606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.60.keytab.
Entry for principal kudu/127.1.49.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.60.keytab.
I20260926 13:44:43.651641 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.60:37869
--webserver_interface=127.1.49.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.60.keytab
--principal=kudu/127.1.49.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:39705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:41479,127.1.49.61:43657,127.1.49.60:37869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.61@KRBTEST.COM for kudu/127.1.49.61@KRBTEST.COM
I20260926 13:44:43.659227 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.648441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:43657 (local address 127.0.0.1:43972)
0926 13:44:43.648684 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.648697 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:43.648774 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:43.649511 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:43.649513 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:43.649676 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:43.649775 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.649781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.651015 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.651022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.651602 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.651611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.652676 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.652679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.652765 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.653387 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:43.653407 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:43.655086 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:43.657451 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.657458 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.657461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.657676 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.658013 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.658016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.658018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.658103 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.658630 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:43.658633 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:43.658739 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 13:44:43.658852 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 13:44:43.659018 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260926 13:44:43.659292 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.648608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.1.49.61:43657)
0926 13:44:43.649044 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.649062 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.649086 (+ 24us) server_negotiation.cc:407] Connection header received
0926 13:44:43.649130 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.649134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.649320 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.649510 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.649879 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.650901 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.651728 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.652545 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.652684 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.655259 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.655269 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.655285 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.655308 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.657306 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.657803 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.657815 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.657816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.657904 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.658206 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.658209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.658211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.658509 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.658665 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.658980 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.659127 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":314,"spinlock_wait_cycles":2944,"thread_start_us":200,"threads_started":1}
I20260926 13:44:43.661960 23606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:44:43.662820 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.662506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37869 (local address 127.0.0.1:53892)
0926 13:44:43.662649 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.662742 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 13:44:43.663056 23555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
I20260926 13:44:43.664104 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.663881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37869 (local address 127.0.0.1:53908)
0926 13:44:43.663990 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.664044 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 13:44:43.664283 23606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:37869: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
I20260926 13:44:43.713606 23606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } attempt: 1
I20260926 13:44:43.714576 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.714246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37869 (local address 127.0.0.1:53914)
0926 13:44:43.714421 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.714492 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 13:44:43.715375 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.715124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37869 (local address 127.0.0.1:53924)
0926 13:44:43.715243 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.715300 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 13:44:43.715508 23606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:37869: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
I20260926 13:44:43.733148 23531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } attempt: 4
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for kudu/127.1.49.61@KRBTEST.COM
I20260926 13:44:43.744374 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.733821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:43657 (local address 127.0.0.1:43988)
0926 13:44:43.734070 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.734085 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:43.734179 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:43.734491 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:43.734493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:43.734668 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:43.734784 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.734789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.735921 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.735923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.736908 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.736920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.738349 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.738352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.738449 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.739048 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:43.739061 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:43.740484 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:43.742628 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.742631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.742632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.742843 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.743161 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.743163 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.743165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.743245 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.743723 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:43.743726 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:43.743912 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 13:44:43.744024 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 13:44:43.744171 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260926 13:44:43.744397 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.733979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.1.49.61:43657)
0926 13:44:43.734145 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.734151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.734173 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:44:43.734265 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.734268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.734322 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.734410 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.734917 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.735742 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.737053 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.738204 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.738336 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.740648 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.740652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.740655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.740685 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.742491 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.742952 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.742956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.742958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.743056 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.743339 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.743343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.743345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.743598 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.743702 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.744099 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.744238 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:44:43.745765 23531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:44:43.746757 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.746493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37869 (local address 127.0.0.1:53934)
0926 13:44:43.746615 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.746681 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 13:44:43.747560 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.747331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37869 (local address 127.0.0.1:53938)
0926 13:44:43.747449 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.747503 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 13:44:43.747726 23531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:37869: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
W20260926 13:44:43.774986 23612 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:44:43.775411 23612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:43.775485 23612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:43.775569 23612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:43.779773 23612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:43.779918 23612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:43.779950 23612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:43.779978 23612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:43.780007 23612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:43.783706 23606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } attempt: 2
I20260926 13:44:43.784718 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.784361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37869 (local address 127.0.0.1:53944)
0926 13:44:43.784550 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.784633 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 13:44:43.785514 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.785285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37869 (local address 127.0.0.1:53958)
0926 13:44:43.785399 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.785457 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 13:44:43.785656 23606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:37869: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
I20260926 13:44:43.786159 23612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:39705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:41479,127.1.49.61:43657,127.1.49.60:37869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.60.keytab
--principal=kudu/127.1.49.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.60:37869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.49.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:43.787290 23612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:43.788422 23612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:43.798269 23619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:43.798518 23612 server_base.cc:1034] running on GCE node
W20260926 13:44:43.798254 23618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:43.798274 23621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:43.799445 23612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:43.800926 23612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:43.802158 23612 hybrid_clock.cc:648] HybridClock initialized: now 1790430283802097 us; error 72 us; skew 500 ppm
I20260926 13:44:43.806967 23531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } attempt: 1
I20260926 13:44:43.808022 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.807592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37869 (local address 127.0.0.1:53974)
0926 13:44:43.807780 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.807930 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 13:44:43.808863 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.808581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37869 (local address 127.0.0.1:53976)
0926 13:44:43.808709 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.808760 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 13:44:43.809046 23531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:37869: Network error: Client connection negotiation failed: client connection to 127.1.49.60:37869: connect: Connection refused (error 111)
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:43.811663 23612 init.cc:377] Logged in from keytab as kudu/127.1.49.60@KRBTEST.COM (short username kudu)
I20260926 13:44:43.812906 23612 webserver.cc:460] Webserver started at http://127.1.49.60:34745/ using document root <none> and password file <none>
I20260926 13:44:43.813587 23612 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:43.813704 23612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:43.813977 23612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:43.816498 23612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/data/instance:
uuid: "3844194ab9b1420c97bde421c5ce6ac2"
format_stamp: "Formatted at 2026-09-26 13:44:43 on dist-test-slave-1wns"
server_key: "c8f15b53366965e94c6b3daaa02fb5ab"
server_key_iv: "19627564666b518d2d9241f6d138684d"
server_key_version: "encryptionkey@0"
I20260926 13:44:43.817181 23612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/wal/instance:
uuid: "3844194ab9b1420c97bde421c5ce6ac2"
format_stamp: "Formatted at 2026-09-26 13:44:43 on dist-test-slave-1wns"
server_key: "c8f15b53366965e94c6b3daaa02fb5ab"
server_key_iv: "19627564666b518d2d9241f6d138684d"
server_key_version: "encryptionkey@0"
I20260926 13:44:43.821770 23612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:44:43.824805 23628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:43.826467 23612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:44:43.826670 23612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/wal
uuid: "3844194ab9b1420c97bde421c5ce6ac2"
format_stamp: "Formatted at 2026-09-26 13:44:43 on dist-test-slave-1wns"
server_key: "c8f15b53366965e94c6b3daaa02fb5ab"
server_key_iv: "19627564666b518d2d9241f6d138684d"
server_key_version: "encryptionkey@0"
I20260926 13:44:43.826830 23612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:43.846572 23612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:43.850404 23612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:43.850670 23612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:43.859002 23612 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.60:37869
I20260926 13:44:43.859359 23680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.60:37869 every 8 connection(s)
I20260926 13:44:43.860404 23612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/data/info.pb
I20260926 13:44:43.863600 23681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:43.867604 23681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:44:43.870249 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 23612
I20260926 13:44:43.870404 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790429983839632-1220-0/minicluster-data/master-2/wal/instance
I20260926 13:44:43.870745 1220 external_mini_cluster.cc:1428] Setting key e2db71791c434fc3664117808a059f81
I20260926 13:44:43.876209 23531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } attempt: 2
I20260926 13:44:43.876860 23606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } attempt: 3
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.60@KRBTEST.COM
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.60@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for kudu/127.1.49.60@KRBTEST.COM
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.61@KRBTEST.COM for kudu/127.1.49.60@KRBTEST.COM
I20260926 13:44:43.886418 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.869761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.1.49.62:41479)
0926 13:44:43.869926 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.869930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.870990 (+ 1060us) server_negotiation.cc:407] Connection header received
0926 13:44:43.872503 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.872507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.872570 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.872686 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.873597 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.874180 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.875328 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.875599 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.875731 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.880920 (+ 5189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.880924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.880929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.880967 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.882985 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.883888 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.883892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.883894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.884022 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.884691 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.884697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.884698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.885041 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.885144 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.886032 (+ 888us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.886204 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:44:43.888583 23685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.871517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.1.49.60:37869)
0926 13:44:43.871999 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.872008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.872044 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:44:43.872252 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.872274 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.872567 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.872776 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.873658 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.874538 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.875398 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.875787 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.876004 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.880482 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.880497 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.880516 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.880548 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.885099 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.885767 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.885774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.885802 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.885913 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.886285 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.886289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.886291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.886876 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.887099 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.887926 (+ 827us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.888107 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"spinlock_wait_cycles":2304,"thread_start_us":126,"threads_started":1}
I20260926 13:44:43.888588 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.870166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:41479 (local address 127.0.0.1:34686)
0926 13:44:43.870778 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.870854 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:43.872245 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:43.872741 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:43.872748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:43.873147 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:43.873436 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.873451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.874318 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.874322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.875189 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.875199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.875776 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.875781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.875999 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.878339 (+ 2340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:43.878361 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:43.880724 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:43.883139 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.883153 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.883168 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.883654 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.884220 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.884227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.884232 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.884549 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.885228 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:43.885238 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:43.885658 (+ 420us) client_negotiation.cc:769] Sending connection context
0926 13:44:43.886850 (+ 1192us) client_negotiation.cc:240] Negotiation successful
0926 13:44:43.887267 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"spinlock_wait_cycles":4992,"thread_start_us":160,"threads_started":1}
I20260926 13:44:43.888958 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.877350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37869 (local address 127.0.0.1:54014)
0926 13:44:43.877578 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.877594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:43.877723 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:43.878220 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:43.878224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:43.878458 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:43.878590 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.878603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.879972 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.879977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.880893 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.880907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.881715 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.881720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.881832 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.882453 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:43.882475 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:43.884302 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:43.887092 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.887096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.887098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.887395 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.887760 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.887763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.887765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.887908 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.888403 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:43.888406 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:43.888513 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 13:44:43.888604 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 13:44:43.888778 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260926 13:44:43.889139 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.876779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37869 (local address 127.0.0.1:54002)
0926 13:44:43.876961 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.876979 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:43.877076 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:43.877698 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:43.877702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:43.877945 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:43.878074 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.878085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.879336 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.879343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.880338 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.880355 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.881057 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.881068 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.881220 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.881711 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:43.881728 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:43.883661 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:43.887136 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.887140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.887142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.887415 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.887979 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.887982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.887984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.888084 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.888551 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:43.888554 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:43.888672 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 13:44:43.888771 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 13:44:43.888889 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 13:44:43.889562 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.877506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.1.49.60:37869)
0926 13:44:43.877957 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.877969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.877983 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:44:43.878016 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.878019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.878060 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.878162 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.878783 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.879769 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.881049 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.881549 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.882059 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.884491 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.884495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.884497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.884547 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.886964 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.887531 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.887534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.887536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.887639 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.888058 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.888064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.888068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.888279 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.889262 (+ 983us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.889341 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.889439 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":377,"spinlock_wait_cycles":13568,"thread_start_us":102,"threads_started":1}
I20260926 13:44:43.889842 23686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.876962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.1.49.60:37869)
0926 13:44:43.877379 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.877399 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.877413 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:44:43.877452 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.877455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.877507 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.878192 (+ 685us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.878301 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.879200 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.880506 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.880906 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.881487 (+ 581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.883933 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.883938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.883941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.883971 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.887003 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.887545 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.887548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.887550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.887672 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.888210 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.888214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.888216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.888429 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.889474 (+ 1045us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.889558 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.889679 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":346,"thread_start_us":138,"threads_started":1}
I20260926 13:44:43.891705 23681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:44:43.894188 23606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6: Bootstrap starting.
I20260926 13:44:43.894213 23531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33: Bootstrap starting.
I20260926 13:44:43.897003 23606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:43.897047 23531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:43.898022 23606 log.cc:824] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:43.898025 23531 log.cc:824] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:43.900439 23531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33: No bootstrap required, opened a new log
I20260926 13:44:43.900452 23606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6: No bootstrap required, opened a new log
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.60@KRBTEST.COM for kudu/127.1.49.61@KRBTEST.COM
I20260926 13:44:43.904851 23531 raft_consensus.cc:348] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } }
I20260926 13:44:43.905179 23531 raft_consensus.cc:729] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41209f6d7014da29ad4999d6dcd9f33, State: Initialized, Role: FOLLOWER
I20260926 13:44:43.905628 23531 consensus_queue.cc:260] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } }
I20260926 13:44:43.905699 23606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } }
I20260926 13:44:43.906054 23606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ada876df0ce4230acb177d205529ac6, State: Initialized, Role: FOLLOWER
I20260926 13:44:43.906488 23606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } }
I20260926 13:44:43.907493 23531 sys_catalog.cc:564] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:43.908263 23606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:43.909137 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.895979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.1.49.62:41479)
0926 13:44:43.896149 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.896154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.896173 (+ 19us) server_negotiation.cc:407] Connection header received
0926 13:44:43.896222 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.896225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.896277 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.896388 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.897487 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.898321 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.899599 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.899973 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.900060 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.904662 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.904667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.904670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.904702 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.906658 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.907305 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.907309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.907311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.907425 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.907784 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.907787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.907789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.908079 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.908246 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.908764 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.908932 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 13:44:43.910065 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.896565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.1.49.61:43657)
0926 13:44:43.896708 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.896713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.897640 (+ 927us) server_negotiation.cc:407] Connection header received
0926 13:44:43.897832 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.897835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.897893 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.897956 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.898692 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.899542 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.900857 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.901191 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.901285 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.905454 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.905459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.905462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.905494 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.907690 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.908425 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.908429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.908431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.908524 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.908905 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.908908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.908910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.909192 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.909332 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.909753 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.909897 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:44:43.910647 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.896316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:43657 (local address 127.0.0.1:43996)
0926 13:44:43.897567 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.897584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:43.897715 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:43.898126 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:43.898131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:43.898326 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:43.898540 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.898550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.899730 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.899733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.900691 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.900713 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.901464 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.901468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.901587 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.902113 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:43.902129 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:43.904939 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:43.907947 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.907951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.907953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.908264 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.908661 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.908664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.908666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.908779 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.909395 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:43.909398 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:43.909524 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 13:44:43.910223 (+ 699us) client_negotiation.cc:240] Negotiation successful
0926 13:44:43.910420 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163}
I20260926 13:44:43.911754 23681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } has no permanent_uuid. Determining permanent_uuid...
W20260926 13:44:43.915501 23707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:43.915863 23707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:43.912559 23691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } } }
I20260926 13:44:43.917084 23691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.61@KRBTEST.COM
W20260926 13:44:43.920928 23713 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:43.921185 23713 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:43.917156 23463 master_runner.cc:186] Error getting master registration for 127.1.49.60:37869: OK, instance_id { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" instance_seqno: 1790430283856780 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:44:43.921736 23463 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:44:43.924367 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.911163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.1.49.61:43657)
0926 13:44:43.911330 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.911336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.912554 (+ 1218us) server_negotiation.cc:407] Connection header received
0926 13:44:43.912603 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.912606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.912658 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.912745 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.913804 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.914608 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.916090 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.916456 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.916539 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.920510 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.920515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.920518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.920571 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.922371 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.922911 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.922916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.922918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.923013 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.923349 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.923353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.923355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.923566 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.923768 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.924036 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.924179 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 13:44:43 dist-test-slave-1wns krb5kdc[23450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430283, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.60@KRBTEST.COM for kudu/127.1.49.60@KRBTEST.COM
I20260926 13:44:43.922539 23538 master_runner.cc:186] Error getting master registration for 127.1.49.60:37869: OK, instance_id { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" instance_seqno: 1790430283856780 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:44:43.926862 23538 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:44:43.927209 23690 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } } }
I20260926 13:44:43.927482 23690 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:44:43.929620 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:43.929832 23686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.912752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.1.49.60:37869)
0926 13:44:43.916154 (+ 3402us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.916160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.916174 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:44:43.916213 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.916216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.916260 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.916337 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.916877 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.917714 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.918854 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.919251 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.919346 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.925781 (+ 6435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.925800 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.925807 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.925839 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.927766 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.928344 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.928347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.928349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.928440 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.928762 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.928766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.928768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.929063 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.929147 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.929498 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.929651 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3305}
I20260926 13:44:43.930281 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.912347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:37869 (local address 127.0.0.1:54022)
0926 13:44:43.915851 (+ 3504us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:43.915870 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:43.915954 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:43.916461 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:43.916464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:43.916662 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:43.916751 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.916758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.917846 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.917851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.918728 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:43.918737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.919542 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.919547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:43.919651 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.923921 (+ 4270us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:43.923943 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:43.925619 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:43.927952 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.927955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.927957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.928232 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.928555 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:43.928560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:43.928562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:43.928659 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:43.929213 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:43.929215 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:43.929314 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 13:44:43.929964 (+ 650us) client_negotiation.cc:240] Negotiation successful
0926 13:44:43.930114 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3386}
I20260926 13:44:43.931425 23505 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a41209f6d7014da29ad4999d6dcd9f33"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34700
I20260926 13:44:43.931744 23505 raft_consensus.cc:482] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 13:44:43.931915 23505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:43.933967 23505 raft_consensus.cc:504] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } }
I20260926 13:44:43.934621 23681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2: Bootstrap starting.
I20260926 13:44:43.935113 23505 leader_election.cc:288] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [CANDIDATE]: Term 1 election: Requested vote from peers 6ada876df0ce4230acb177d205529ac6 (127.1.49.61:43657), 3844194ab9b1420c97bde421c5ce6ac2 (127.1.49.60:37869)
I20260926 13:44:43.935638 23580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a41209f6d7014da29ad4999d6dcd9f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6ada876df0ce4230acb177d205529ac6"
I20260926 13:44:43.935957 23580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:43.937475 23580 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41209f6d7014da29ad4999d6dcd9f33 in term 1.
I20260926 13:44:43.938083 23479 leader_election.cc:302] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ada876df0ce4230acb177d205529ac6, a41209f6d7014da29ad4999d6dcd9f33; no voters:
I20260926 13:44:43.938472 23690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:43.938748 23690 raft_consensus.cc:686] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [term 1 LEADER]: Becoming Leader. State: Replica: a41209f6d7014da29ad4999d6dcd9f33, State: Running, Role: LEADER
I20260926 13:44:43.939162 23690 consensus_queue.cc:237] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } }
I20260926 13:44:43.940845 23681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:43.941831 23681 log.cc:824] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:43.944213 23721 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a41209f6d7014da29ad4999d6dcd9f33. Latest consensus state: current_term: 1 leader_uuid: "a41209f6d7014da29ad4999d6dcd9f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } } }
I20260926 13:44:43.944425 23721 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:43.956143 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.938417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.1.49.61:43657)
0926 13:44:43.938546 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.938552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.945134 (+ 6582us) server_negotiation.cc:407] Connection header received
0926 13:44:43.945302 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.945307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.945373 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.945462 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.946995 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.947704 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.949173 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.949567 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.949677 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.951866 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.951871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.951874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.951912 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.953894 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.954492 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.954498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.954501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.954602 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.954923 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.954927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.954929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.955152 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.955345 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.955580 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.955715 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:44:43.958431 23725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:43.959517 23725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:43.963502 23681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2: No bootstrap required, opened a new log
I20260926 13:44:43.965678 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.940248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.1.49.62:41479)
0926 13:44:43.940380 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.940385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.948127 (+ 7742us) server_negotiation.cc:407] Connection header received
0926 13:44:43.948354 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.948358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.948430 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.948492 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.949628 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.950441 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.957095 (+ 6654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.957510 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.957599 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.961149 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.961153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.961157 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.961188 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.963267 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.963873 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.963877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.963880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.963973 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.964290 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.964294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.964296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.964549 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.964683 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.965349 (+ 666us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.965492 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:44:43.972860 23580 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6 [term 1 FOLLOWER]: Refusing update from remote peer a41209f6d7014da29ad4999d6dcd9f33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:43.973661 23721 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:43.981058 23681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } }
I20260926 13:44:43.982430 23691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a41209f6d7014da29ad4999d6dcd9f33. Latest consensus state: current_term: 1 leader_uuid: "a41209f6d7014da29ad4999d6dcd9f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } } }
I20260926 13:44:43.982591 23691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:44:43.982823 23681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3844194ab9b1420c97bde421c5ce6ac2, State: Initialized, Role: FOLLOWER
I20260926 13:44:43.985759 23686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:43.945673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.1.49.60:37869)
0926 13:44:43.945805 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:43.945810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:43.956048 (+ 10238us) server_negotiation.cc:407] Connection header received
0926 13:44:43.956417 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:43.956421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:43.956488 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:43.956549 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:43.957538 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.958383 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.959685 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:43.960087 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:43.960219 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:43.974912 (+ 14693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:43.974918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:43.974921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:43.974966 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:43.977236 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.981949 (+ 4713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.981955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.981957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.982082 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:43.983525 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:43.983529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:43.983531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:43.983823 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:43.983994 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:43.985413 (+ 1419us) server_negotiation.cc:299] Negotiation successful
0926 13:44:43.985575 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:44:43.986263 23681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } }
I20260926 13:44:43.986930 23690 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a41209f6d7014da29ad4999d6dcd9f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } } }
I20260926 13:44:43.987087 23690 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:43.987823 23691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a41209f6d7014da29ad4999d6dcd9f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } } }
I20260926 13:44:43.987960 23691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:44:43.988409 23727 mvcc.cc:205] Tried to move back new op lower bound from 7333602443140362240 to 7333602443018973184. Current Snapshot: MvccSnapshot[applied={T|T < 7333602443140362240}]
I20260926 13:44:43.990943 23728 mvcc.cc:205] Tried to move back new op lower bound from 7333602443140362240 to 7333602443018973184. Current Snapshot: MvccSnapshot[applied={T|T < 7333602443140362240}]
I20260926 13:44:43.991298 23655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a41209f6d7014da29ad4999d6dcd9f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3844194ab9b1420c97bde421c5ce6ac2"
I20260926 13:44:43.991526 23654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:43.991849 23681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:43.993218 23654 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2 [term 1 FOLLOWER]: Refusing update from remote peer a41209f6d7014da29ad4999d6dcd9f33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:43.994230 23721 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:43.995368 23655 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41209f6d7014da29ad4999d6dcd9f33 in term 1.
I20260926 13:44:44.000350 23725 catalog_manager.cc:1347] Generated new cluster ID: f87823a51ab34905b3d0c63e9cf58c01
I20260926 13:44:44.000453 23725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:44.005584 23738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a41209f6d7014da29ad4999d6dcd9f33. Latest consensus state: current_term: 1 leader_uuid: "a41209f6d7014da29ad4999d6dcd9f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } } }
I20260926 13:44:44.005769 23738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:44:44.005887 23738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a41209f6d7014da29ad4999d6dcd9f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } } }
I20260926 13:44:44.005954 23738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:44:44.017104 23731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a41209f6d7014da29ad4999d6dcd9f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } } }
I20260926 13:44:44.017503 23731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 13:44:44.025345 23747 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:44.025463 23747 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:44.028330 23690 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a41209f6d7014da29ad4999d6dcd9f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41209f6d7014da29ad4999d6dcd9f33" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 41479 } } peers { permanent_uuid: "6ada876df0ce4230acb177d205529ac6" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 43657 } } peers { permanent_uuid: "3844194ab9b1420c97bde421c5ce6ac2" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 37869 } } }
I20260926 13:44:44.028508 23690 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:44.032934 23725 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:44.033697 23744 mvcc.cc:205] Tried to move back new op lower bound from 7333602443140362240 to 7333602443018973184. Current Snapshot: MvccSnapshot[applied={T|T < 7333602443140362240}]
I20260926 13:44:44.034446 23725 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:44.059098 23725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a41209f6d7014da29ad4999d6dcd9f33: Generated new TSK 0
I20260926 13:44:44.060395 23725 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:44:44.128736 23686 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:44.125057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.1.49.60:37869)
0926 13:44:44.125179 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:44.125185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:44.127192 (+ 2007us) server_negotiation.cc:407] Connection header received
0926 13:44:44.127453 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:44.127457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:44.127517 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:44.127580 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:44.128568 (+ 988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54038: BlockingRecv error: recv got EOF from 127.0.0.1:54038 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 13:44:44.130833 23611 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:44.124510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44016 (local address 127.1.49.61:43657)
0926 13:44:44.124667 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:44.124672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:44.129649 (+ 4977us) server_negotiation.cc:407] Connection header received
0926 13:44:44.129875 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:44.129878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:44.129941 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:44.130000 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:44.130692 (+ 692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44016: BlockingRecv error: recv got EOF from 127.0.0.1:44016 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 13:44:44.133127 23535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:44.126463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.1.49.62:41479)
0926 13:44:44.126609 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:44.126614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:44.131715 (+ 5101us) server_negotiation.cc:407] Connection header received
0926 13:44:44.132122 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:44.132126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:44.132188 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:44.132289 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:44.132949 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34708: BlockingRecv error: recv got EOF from 127.0.0.1:34708 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 13:44:44.139264 23611 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:44.137094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.1.49.61:43657)
0926 13:44:44.137245 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:44.137251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:44.137971 (+ 720us) server_negotiation.cc:407] Connection header received
0926 13:44:44.138266 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:44.138271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:44.138366 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:44.138437 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:44.139113 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44020: BlockingRecv error: recv got EOF from 127.0.0.1:44020 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 13:44:44.147605 23611 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:44.144851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.1.49.61:43657)
0926 13:44:44.145044 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:44.145050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:44.146199 (+ 1149us) server_negotiation.cc:407] Connection header received
0926 13:44:44.146470 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:44.146474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:44.146542 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:44.146607 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:44.147463 (+ 856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44034: BlockingRecv error: recv got EOF from 127.0.0.1:44034 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260926 13:44:44.154994 23611 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:44.152923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44042 (local address 127.1.49.61:43657)
0926 13:44:44.153066 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:44.153071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:44.153798 (+ 727us) server_negotiation.cc:407] Connection header received
0926 13:44:44.154060 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:44.154065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:44.154129 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:44.154194 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:44.154860 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44042: BlockingRecv error: recv got EOF from 127.0.0.1:44042 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 13:44:44.164287 23611 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:44.162122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.1.49.61:43657)
0926 13:44:44.162299 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:44.162304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:44.163095 (+ 791us) server_negotiation.cc:407] Connection header received
0926 13:44:44.163357 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:44.163361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:44.163438 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:44.163497 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:44.164158 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44044: BlockingRecv error: recv got EOF from 127.0.0.1:44044 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260926 13:44:44.177621 23611 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:44.175522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.1.49.61:43657)
0926 13:44:44.175668 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:44.175673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:44.176361 (+ 688us) server_negotiation.cc:407] Connection header received
0926 13:44:44.176618 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:44.176624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:44.176703 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:44.176792 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:44.177476 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44050: BlockingRecv error: recv got EOF from 127.0.0.1:44050 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 13:44:44.198305 23611 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:44.196795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.1.49.61:43657)
0926 13:44:44.196942 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:44.196947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:44.197306 (+ 359us) server_negotiation.cc:407] Connection header received
0926 13:44:44.197493 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:44.197504 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:44.197558 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:44.197658 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:44.198154 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44062: BlockingRecv error: recv got EOF from 127.0.0.1:44062 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 13:44:44.234234 23611 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:44.232825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44074 (local address 127.1.49.61:43657)
0926 13:44:44.233012 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:44.233016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:44.233237 (+ 221us) server_negotiation.cc:407] Connection header received
0926 13:44:44.233378 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:44.233381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:44.233430 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:44.233508 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:44.234102 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44074: BlockingRecv error: recv got EOF from 127.0.0.1:44074 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 13:44:44.302467 23611 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:44.300873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.1.49.61:43657)
0926 13:44:44.301058 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:44.301064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:44.301307 (+ 243us) server_negotiation.cc:407] Connection header received
0926 13:44:44.301514 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:44.301526 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:44.301581 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:44.301664 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:44.302337 (+ 673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44082: BlockingRecv error: recv got EOF from 127.0.0.1:44082 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 13:44:44.434604 23611 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:44.433164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.1.49.61:43657)
0926 13:44:44.433342 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:44.433347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:44.433574 (+ 227us) server_negotiation.cc:407] Connection header received
0926 13:44:44.433769 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:44.433773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:44.433842 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:44.433946 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:44.434465 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44096: BlockingRecv error: recv got EOF from 127.0.0.1:44096 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 13:44:44.694497 23611 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:44.693414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44108 (local address 127.1.49.61:43657)
0926 13:44:44.693589 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:44.693593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:44.693609 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:44:44.693700 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:44.693703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:44.693751 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:44.693828 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:44.694370 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44108: BlockingRecv error: recv got EOF from 127.0.0.1:44108 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 13:44:44.922761 23713 catalog_manager.cc:1259] Loaded cluster ID: f87823a51ab34905b3d0c63e9cf58c01
I20260926 13:44:44.922905 23713 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6: loading cluster ID for follower catalog manager: success
I20260926 13:44:44.924829 23713 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6: acquiring CA information for follower catalog manager: success
I20260926 13:44:44.926110 23713 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ada876df0ce4230acb177d205529ac6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:44:45.026773 23747 catalog_manager.cc:1259] Loaded cluster ID: f87823a51ab34905b3d0c63e9cf58c01
I20260926 13:44:45.026858 23747 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2: loading cluster ID for follower catalog manager: success
I20260926 13:44:45.028721 23747 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2: acquiring CA information for follower catalog manager: success
I20260926 13:44:45.029982 23747 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3844194ab9b1420c97bde421c5ce6ac2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:44:45.214756 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:45.209719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.1.49.61:43657)
0926 13:44:45.210005 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:45.210009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:45.210230 (+ 221us) server_negotiation.cc:407] Connection header received
0926 13:44:45.210448 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:45.210456 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:45.210587 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:45.210695 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:45.211108 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:45.211731 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:45.212843 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:45.213080 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:45.213174 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:45.213614 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:45.214123 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:45.214234 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:45.214461 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 13:44:45.214528 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":119,"threads_started":1}
I20260926 13:44:45.222864 23822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:45.218426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54044 (local address 127.1.49.60:37869)
0926 13:44:45.218738 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:45.218745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:45.218770 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:44:45.219056 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:45.219060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:45.219190 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:45.219329 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:45.219589 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:45.220238 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:45.221222 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:45.221481 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:45.221601 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:45.221936 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:45.222427 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:45.222585 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:45.222624 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 13:44:45.222686 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260926 13:44:45.231331 23822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:45.226483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.1.49.60:37869)
0926 13:44:45.226628 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:45.226632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:45.226953 (+ 321us) server_negotiation.cc:407] Connection header received
0926 13:44:45.227213 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:45.227217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:45.227283 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:45.227342 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:45.227842 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:45.228501 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:45.229905 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:45.230297 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:45.230407 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:45.230779 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:45.230906 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:45.230979 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:45.231083 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 13:44:45.231141 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:44:45.231559 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:45.226408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44126 (local address 127.1.49.61:43657)
0926 13:44:45.226559 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:45.226564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:45.227136 (+ 572us) server_negotiation.cc:407] Connection header received
0926 13:44:45.227305 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:45.227309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:45.227384 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:45.227467 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:45.227770 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:45.228686 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:45.230128 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:45.230524 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:45.230657 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:45.230983 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:45.231074 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:45.231150 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:45.231310 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 13:44:45.231377 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 13:44:45.231930 23828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:45.226413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34714 (local address 127.1.49.62:41479)
0926 13:44:45.226821 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:45.226828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:45.227169 (+ 341us) server_negotiation.cc:407] Connection header received
0926 13:44:45.227445 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:45.227452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:45.227636 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:45.227774 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:45.228097 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:45.228937 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:45.230030 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:45.230316 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:45.230452 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:45.230963 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:45.231385 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:45.231473 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:45.231673 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 13:44:45.231741 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":77,"threads_started":1}
I20260926 13:44:45.236093 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 23463
I20260926 13:44:45.242628 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 23538
I20260926 13:44:45.248731 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 23612
2026-09-26T13:44:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3372 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:45 dist-test-slave-1wns krb5kdc[23833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:45 dist-test-slave-1wns krb5kdc[23833](info): set up 2 sockets
Sep 26 13:44:45 dist-test-slave-1wns krb5kdc[23833](info): commencing operation
krb5kdc: starting...
W20260926 13:44:46.356945 1220 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/test-user.keytab.
Sep 26 13:44:46 dist-test-slave-1wns krb5kdc[23833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.49.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
Entry for principal kudu/127.1.49.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab.
I20260926 13:44:46.495707 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:40413
--webserver_interface=127.1.49.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:40413,127.1.49.61:35593,127.1.49.60:43953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:46.619313 23846 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:44:46.619673 23846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:46.619699 23846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:46.619793 23846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:46.624163 23846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:46.624248 23846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:46.624269 23846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:46.624285 23846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:46.624301 23846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:46.630259 23846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:40413,127.1.49.61:35593,127.1.49.60:43953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.62.keytab
--principal=kudu/127.1.49.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.62:40413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.49.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:46.631557 23846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:46.632742 23846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:46.642412 23851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:46.642349 23852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:46.642396 23846 server_base.cc:1034] running on GCE node
W20260926 13:44:46.642344 23854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:46.643397 23846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:46.644675 23846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:46.645852 23846 hybrid_clock.cc:648] HybridClock initialized: now 1790430286645825 us; error 43 us; skew 500 ppm
Sep 26 13:44:46 dist-test-slave-1wns krb5kdc[23833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430286, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:46.649101 23846 init.cc:377] Logged in from keytab as kudu/127.1.49.62@KRBTEST.COM (short username kudu)
I20260926 13:44:46.650187 23846 webserver.cc:460] Webserver started at http://127.1.49.62:45601/ using document root <none> and password file <none>
I20260926 13:44:46.650840 23846 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:46.650915 23846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:46.651167 23846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:46.653539 23846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/data/instance:
uuid: "99c165160ad04b9a9591454f1c5ce3d2"
format_stamp: "Formatted at 2026-09-26 13:44:46 on dist-test-slave-1wns"
server_key: "d8f94b6b627d34936fcc439a3cf5ca24"
server_key_iv: "d200f6ec4f3d13ad8e14bcc293f3afae"
server_key_version: "encryptionkey@0"
I20260926 13:44:46.654268 23846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/wal/instance:
uuid: "99c165160ad04b9a9591454f1c5ce3d2"
format_stamp: "Formatted at 2026-09-26 13:44:46 on dist-test-slave-1wns"
server_key: "d8f94b6b627d34936fcc439a3cf5ca24"
server_key_iv: "d200f6ec4f3d13ad8e14bcc293f3afae"
server_key_version: "encryptionkey@0"
I20260926 13:44:46.658581 23846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 13:44:46.661605 23861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:46.663144 23846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 13:44:46.663329 23846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/wal
uuid: "99c165160ad04b9a9591454f1c5ce3d2"
format_stamp: "Formatted at 2026-09-26 13:44:46 on dist-test-slave-1wns"
server_key: "d8f94b6b627d34936fcc439a3cf5ca24"
server_key_iv: "d200f6ec4f3d13ad8e14bcc293f3afae"
server_key_version: "encryptionkey@0"
I20260926 13:44:46.663483 23846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:46.676856 23846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:46.680888 23846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:46.681149 23846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:46.690460 23846 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.62:40413
I20260926 13:44:46.690804 23913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.62:40413 every 8 connection(s)
I20260926 13:44:46.691704 23846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:46.695092 23914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:46.697633 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 23846
I20260926 13:44:46.697796 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-0/wal/instance
I20260926 13:44:46.698145 1220 external_mini_cluster.cc:1428] Setting key f2d3614148571eb945e669b016dfe00e
I20260926 13:44:46.700958 23914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.49.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.61@KRBTEST.COM" created.
Sep 26 13:44:46 dist-test-slave-1wns krb5kdc[23833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430286, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:46.719669 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.703984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.1.49.62:40413)
0926 13:44:46.704567 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:46.704588 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:46.704856 (+ 268us) server_negotiation.cc:407] Connection header received
0926 13:44:46.706191 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:46.706197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:46.706434 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:46.706655 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:46.707791 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.708940 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.710119 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.710662 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.710853 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:46.714088 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:46.714098 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:46.714117 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:46.714142 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:46.716150 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:46.716709 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:46.716723 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:46.716725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:46.716841 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:46.717173 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:46.717176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:46.717178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:46.717626 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:46.717775 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:46.718184 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 13:44:46.718541 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":338,"thread_start_us":178,"threads_started":1}
I20260926 13:44:46.719669 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.703986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:40413 (local address 127.0.0.1:60534)
0926 13:44:46.704687 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.704730 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:46.705806 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:46.706652 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:46.706658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:46.707313 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:46.707616 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:46.707632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.709098 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.709102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:46.709979 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:46.709998 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.710868 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.710872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:46.711000 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:46.711690 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:46.711708 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:46.713900 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:46.716284 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:46.716296 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:46.716310 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:46.716585 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:46.716943 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:46.716948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:46.716950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:46.717066 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:46.717753 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:46.717759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:46.717910 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 13:44:46.718112 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 13:44:46.718595 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":166,"spinlock_wait_cycles":11008,"thread_start_us":198,"threads_started":1}
I20260926 13:44:46.722720 23914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:44:46.723758 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.723441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:35593 (local address 127.0.0.1:46018)
0926 13:44:46.723597 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.723680 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 13:44:46.724249 23865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.49.61:35593: connect: Connection refused (error 111)
I20260926 13:44:46.725529 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.725268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:35593 (local address 127.0.0.1:46022)
0926 13:44:46.725388 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.725447 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 13:44:46.725710 23914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.61:35593: Network error: Client connection negotiation failed: client connection to 127.1.49.61:35593: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.61.keytab.
Entry for principal kudu/127.1.49.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.61.keytab.
I20260926 13:44:46.728583 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.61:35593
--webserver_interface=127.1.49.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.61.keytab
--principal=kudu/127.1.49.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:40413,127.1.49.61:35593,127.1.49.60:43953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:44:46.775007 23914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } attempt: 1
I20260926 13:44:46.776126 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.775707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:35593 (local address 127.0.0.1:46032)
0926 13:44:46.775976 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.776043 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 13:44:46.776993 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.776764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:35593 (local address 127.0.0.1:46040)
0926 13:44:46.776874 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.776933 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 13:44:46.777122 23914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.61:35593: Network error: Client connection negotiation failed: client connection to 127.1.49.61:35593: connect: Connection refused (error 111)
I20260926 13:44:46.845337 23914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } attempt: 2
I20260926 13:44:46.846241 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.845921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:35593 (local address 127.0.0.1:46042)
0926 13:44:46.846082 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.846158 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 13:44:46.847028 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.846751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:35593 (local address 127.0.0.1:46054)
0926 13:44:46.846917 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.846963 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.61:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260926 13:44:46.847165 23914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.61:35593: Network error: Client connection negotiation failed: client connection to 127.1.49.61:35593: connect: Connection refused (error 111)
W20260926 13:44:46.850661 23921 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:44:46.850989 23921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:46.851058 23921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:46.851155 23921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:46.855726 23921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:46.855885 23921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:46.855912 23921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:46.855947 23921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:46.855983 23921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:46.861882 23921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:40413,127.1.49.61:35593,127.1.49.60:43953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.61.keytab
--principal=kudu/127.1.49.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.61:35593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.49.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:46.863133 23921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:46.864319 23921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:46.874351 23926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:46.874351 23927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:46.874444 23929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:46.874577 23921 server_base.cc:1034] running on GCE node
I20260926 13:44:46.875507 23921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:46.876852 23921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:46.878062 23921 hybrid_clock.cc:648] HybridClock initialized: now 1790430286878031 us; error 44 us; skew 500 ppm
Sep 26 13:44:46 dist-test-slave-1wns krb5kdc[23833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430286, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:46.881558 23921 init.cc:377] Logged in from keytab as kudu/127.1.49.61@KRBTEST.COM (short username kudu)
I20260926 13:44:46.882614 23921 webserver.cc:460] Webserver started at http://127.1.49.61:35233/ using document root <none> and password file <none>
I20260926 13:44:46.883219 23921 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:46.883298 23921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:46.883492 23921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:46.885896 23921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/data/instance:
uuid: "efd6bc047dc2498884353f8081976bef"
format_stamp: "Formatted at 2026-09-26 13:44:46 on dist-test-slave-1wns"
server_key: "4525e5aabc07ffc75d620f4ca07b8082"
server_key_iv: "f2481e46b44f4d64581565e38c4697e3"
server_key_version: "encryptionkey@0"
I20260926 13:44:46.886639 23921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/wal/instance:
uuid: "efd6bc047dc2498884353f8081976bef"
format_stamp: "Formatted at 2026-09-26 13:44:46 on dist-test-slave-1wns"
server_key: "4525e5aabc07ffc75d620f4ca07b8082"
server_key_iv: "f2481e46b44f4d64581565e38c4697e3"
server_key_version: "encryptionkey@0"
I20260926 13:44:46.891153 23921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 13:44:46.894179 23936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:46.895757 23921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:44:46.895965 23921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/wal
uuid: "efd6bc047dc2498884353f8081976bef"
format_stamp: "Formatted at 2026-09-26 13:44:46 on dist-test-slave-1wns"
server_key: "4525e5aabc07ffc75d620f4ca07b8082"
server_key_iv: "f2481e46b44f4d64581565e38c4697e3"
server_key_version: "encryptionkey@0"
I20260926 13:44:46.896090 23921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:46.922497 23921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:46.926451 23921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:46.926661 23921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:46.935225 23921 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.61:35593
I20260926 13:44:46.935591 23988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.61:35593 every 8 connection(s)
I20260926 13:44:46.936403 23921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/data/info.pb
I20260926 13:44:46.938373 23914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } attempt: 3
I20260926 13:44:46.939944 23989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:46.944088 23989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:44:46.946673 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 23921
I20260926 13:44:46.946851 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-1/wal/instance
I20260926 13:44:46.947206 1220 external_mini_cluster.cc:1428] Setting key 6f0fcf80962dd5ed774825668a51aaa8
Sep 26 13:44:46 dist-test-slave-1wns krb5kdc[23833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430286, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for kudu/127.1.49.61@KRBTEST.COM
Sep 26 13:44:46 dist-test-slave-1wns krb5kdc[23833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430286, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.61@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:46.953078 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.938925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:35593 (local address 127.0.0.1:46070)
0926 13:44:46.939123 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.939144 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:46.939260 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:46.942041 (+ 2781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:46.942045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:46.942292 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:46.942445 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:46.942456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.943892 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.943896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:46.944579 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:46.944591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.945384 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.945389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:46.945510 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:46.946145 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:46.946165 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:46.948021 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:46.950710 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:46.950714 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:46.950717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:46.950964 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:46.951348 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:46.951352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:46.951354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:46.951495 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:46.952468 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:46.952472 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:46.952601 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 13:44:46.952717 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 13:44:46.952891 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 13:44:46.954658 23991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.939828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.1.49.61:35593)
0926 13:44:46.940393 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:46.940416 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:46.940457 (+ 41us) server_negotiation.cc:407] Connection header received
0926 13:44:46.941207 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:46.941225 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:46.941597 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:46.942090 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:46.942608 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.943698 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.944730 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.945233 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.945424 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:46.948204 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:46.948209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:46.948219 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:46.948248 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:46.950585 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:46.951091 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:46.951104 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:46.951110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:46.951234 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:46.951619 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:46.951626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:46.951631 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:46.952317 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:46.952480 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:46.952938 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 13:44:46.953335 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":157,"threads_started":1}
I20260926 13:44:46.957048 23914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:44:46.958026 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.957663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45026)
0926 13:44:46.957883 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.957939 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
WARNING: no policy specified for kudu/127.1.49.60@KRBTEST.COM; defaulting to no policy
I20260926 13:44:46.958352 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.945996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.1.49.62:40413)
0926 13:44:46.946141 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:46.946145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:46.946501 (+ 356us) server_negotiation.cc:407] Connection header received
0926 13:44:46.946698 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:46.946701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:46.946780 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:46.946840 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:46.947949 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.948785 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.950030 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.950830 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.950978 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:46.953880 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:46.953885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:46.953888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:46.953916 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:46.956336 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:46.956856 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:46.956862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:46.956867 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:46.956970 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:46.957369 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:46.957372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:46.957373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:46.957594 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:46.957712 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:46.958071 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 13:44:46.958202 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:44:46.958395 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.945918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:40413 (local address 127.0.0.1:60540)
0926 13:44:46.946366 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.946425 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:46.946589 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:46.947058 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:46.947066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:46.947568 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:46.947778 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:46.947787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.948967 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.948973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:46.949879 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:46.949891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.950982 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.950986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:46.951151 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:46.951906 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:46.951919 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:46.953722 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:46.956465 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:46.956477 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:46.956482 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:46.956732 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:46.957121 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:46.957126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:46.957129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:46.957241 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:46.957710 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:46.957717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:46.957846 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 13:44:46.958002 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 13:44:46.958203 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":164,"threads_started":1}
I20260926 13:44:46.958940 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.958660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45038)
0926 13:44:46.958828 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.958886 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260926 13:44:46.959842 23914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:43953: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
I20260926 13:44:46.960654 23989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.49.60@KRBTEST.COM" created.
Sep 26 13:44:46 dist-test-slave-1wns krb5kdc[23833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430286, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.61@KRBTEST.COM for kudu/127.1.49.61@KRBTEST.COM
I20260926 13:44:46.971127 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.961243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:35593 (local address 127.0.0.1:46082)
0926 13:44:46.961376 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.961448 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:46.961554 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:46.961803 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:46.961808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:46.961965 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:46.962068 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:46.962073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.963296 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.963300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:46.964005 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:46.964016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.964703 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.964706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:46.964785 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:46.965430 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:46.965444 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:46.967036 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:46.969390 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:46.969397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:46.969401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:46.969634 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:46.969966 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:46.969972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:46.969974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:46.970071 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:46.970498 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:46.970501 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:46.970603 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 13:44:46.970719 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 13:44:46.970908 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 13:44:46.971127 23991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.961306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.1.49.61:35593)
0926 13:44:46.961413 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:46.961418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:46.961550 (+ 132us) server_negotiation.cc:407] Connection header received
0926 13:44:46.961637 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:46.961644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:46.961687 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:46.961763 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:46.962188 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.963129 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.964139 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:46.964551 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:46.964679 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:46.967222 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:46.967225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:46.967227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:46.967250 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:46.969128 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:46.969763 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:46.969769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:46.969773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:46.969863 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:46.970167 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:46.970173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:46.970175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:46.970392 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:46.970512 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:46.970779 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 13:44:46.970991 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":40}
I20260926 13:44:46.972118 23989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:44:46.972999 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.972673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45048)
0926 13:44:46.972830 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.972926 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 13:44:46.973783 23937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
I20260926 13:44:46.974882 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.974582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45050)
0926 13:44:46.974712 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.974769 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 13:44:46.975096 23989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:43953: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.49.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.60.keytab.
Entry for principal kudu/127.1.49.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.60.keytab.
I20260926 13:44:46.980315 1220 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7nlTrX/build/debug/bin/kudu
/tmp/dist-test-task7nlTrX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.60:43953
--webserver_interface=127.1.49.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.60.keytab
--principal=kudu/127.1.49.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.49.20:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.49.62:40413,127.1.49.61:35593,127.1.49.60:43953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:44:46.991073 23914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } attempt: 1
I20260926 13:44:46.992069 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.991689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45056)
0926 13:44:46.991890 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.991969 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 13:44:46.992843 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:46.992604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45066)
0926 13:44:46.992717 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:46.992771 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 13:44:46.992982 23914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:43953: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
I20260926 13:44:47.024394 23989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } attempt: 1
I20260926 13:44:47.025475 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.025065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45074)
0926 13:44:47.025237 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.025389 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 13:44:47.026480 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.026144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45084)
0926 13:44:47.026290 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.026416 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 13:44:47.026677 23989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:43953: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
I20260926 13:44:47.068161 23914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } attempt: 2
I20260926 13:44:47.069351 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.068896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45088)
0926 13:44:47.069117 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.069221 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 13:44:47.070375 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.070113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45092)
0926 13:44:47.070254 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.070308 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 13:44:47.070717 23914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:43953: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
I20260926 13:44:47.094894 23989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } attempt: 2
I20260926 13:44:47.095912 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.095494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45094)
0926 13:44:47.095693 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.095767 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 13:44:47.096712 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.096473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45108)
0926 13:44:47.096590 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.096647 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 13:44:47.096851 23989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:43953: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
W20260926 13:44:47.108669 23996 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:44:47.109043 23996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:47.109073 23996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:47.109211 23996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:47.114082 23996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:47.114182 23996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:47.114204 23996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:47.114223 23996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:47.114243 23996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:47.120821 23996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.49.20:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.49.62:40413,127.1.49.61:35593,127.1.49.60:43953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/krb5kdc/kudu_127.1.49.60.keytab
--principal=kudu/127.1.49.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.49.60:43953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.49.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:47.122332 23996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:47.123605 23996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:47.133427 24002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:47.133404 24004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:47.133772 23996 server_base.cc:1034] running on GCE node
W20260926 13:44:47.133409 24001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:47.134485 23996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:47.135754 23996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:47.137029 23996 hybrid_clock.cc:648] HybridClock initialized: now 1790430287137010 us; error 40 us; skew 500 ppm
Sep 26 13:44:47 dist-test-slave-1wns krb5kdc[23833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430287, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:47.140645 23996 init.cc:377] Logged in from keytab as kudu/127.1.49.60@KRBTEST.COM (short username kudu)
I20260926 13:44:47.141813 23996 webserver.cc:460] Webserver started at http://127.1.49.60:46881/ using document root <none> and password file <none>
I20260926 13:44:47.142514 23996 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:47.142634 23996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:47.142920 23996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:47.145371 23996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/data/instance:
uuid: "b3b3a80d23704337962d4c9c5f7917f3"
format_stamp: "Formatted at 2026-09-26 13:44:47 on dist-test-slave-1wns"
server_key: "f50edfba82ed9ffe89dd26e163917472"
server_key_iv: "14bd8cde472f42144faaa1f6ae2c6c56"
server_key_version: "encryptionkey@0"
I20260926 13:44:47.146080 23996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/wal/instance:
uuid: "b3b3a80d23704337962d4c9c5f7917f3"
format_stamp: "Formatted at 2026-09-26 13:44:47 on dist-test-slave-1wns"
server_key: "f50edfba82ed9ffe89dd26e163917472"
server_key_iv: "14bd8cde472f42144faaa1f6ae2c6c56"
server_key_version: "encryptionkey@0"
I20260926 13:44:47.150802 23996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 13:44:47.153951 24011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:47.155658 23996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:44:47.155898 23996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/data,/tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/wal
uuid: "b3b3a80d23704337962d4c9c5f7917f3"
format_stamp: "Formatted at 2026-09-26 13:44:47 on dist-test-slave-1wns"
server_key: "f50edfba82ed9ffe89dd26e163917472"
server_key_iv: "14bd8cde472f42144faaa1f6ae2c6c56"
server_key_version: "encryptionkey@0"
I20260926 13:44:47.156095 23996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:47.169921 23914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } attempt: 3
I20260926 13:44:47.170902 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.170538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45118)
0926 13:44:47.170704 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.170777 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 13:44:47.171756 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.171497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45124)
0926 13:44:47.171619 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.171689 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 13:44:47.172025 23914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:43953: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
I20260926 13:44:47.177681 23996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:47.181766 23996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:47.182024 23996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:47.188047 23989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } attempt: 3
I20260926 13:44:47.189191 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.188656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45128)
0926 13:44:47.188940 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.189052 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 13:44:47.190191 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.189874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45136)
0926 13:44:47.190060 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.190121 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260926 13:44:47.190464 23989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.49.60:43953: Network error: Client connection negotiation failed: client connection to 127.1.49.60:43953: connect: Connection refused (error 111)
I20260926 13:44:47.190868 23996 rpc_server.cc:307] RPC server started. Bound to: 127.1.49.60:43953
I20260926 13:44:47.191223 24063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.49.60:43953 every 8 connection(s)
I20260926 13:44:47.192556 23996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/data/info.pb
I20260926 13:44:47.195991 24064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:47.198328 1220 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7nlTrX/build/debug/bin/kudu as pid 23996
I20260926 13:44:47.198565 1220 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7nlTrX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790429983839632-1220-0/minicluster-data/master-2/wal/instance
I20260926 13:44:47.198899 1220 external_mini_cluster.cc:1428] Setting key df24f590a8c7b5d4a3f70ccb49bb5e58
I20260926 13:44:47.200726 24064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:44:47 dist-test-slave-1wns krb5kdc[23833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430287, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.60@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
Sep 26 13:44:47 dist-test-slave-1wns krb5kdc[23833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.60@KRBTEST.COM
I20260926 13:44:47.215027 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.202710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.1.49.62:40413)
0926 13:44:47.202854 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.202859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.203411 (+ 552us) server_negotiation.cc:407] Connection header received
0926 13:44:47.203581 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.203584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.203683 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.203781 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.204651 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.205318 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.206810 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.207191 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.207305 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.210373 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:47.210381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:47.210396 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:47.210458 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:47.212416 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.213024 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.213028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.213030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.213126 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.213648 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.213651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.213652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.213955 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:47.214068 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:47.214673 (+ 605us) server_negotiation.cc:299] Negotiation successful
0926 13:44:47.214850 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:44:47.216001 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.202558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.62:40413 (local address 127.0.0.1:60546)
0926 13:44:47.203238 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.203300 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:47.203444 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:47.203915 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:47.203921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:47.204300 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:47.204493 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:47.204506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.205444 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.205448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:47.206649 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:47.206659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.207376 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.207382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:47.207531 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.208217 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:47.208241 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:47.210185 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:47.212578 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:47.212586 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:47.212596 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:47.212914 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:47.213226 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:47.213229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:47.213230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:47.213540 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:47.214085 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:47.214093 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:47.214426 (+ 333us) client_negotiation.cc:769] Sending connection context
0926 13:44:47.214592 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 13:44:47.214962 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":128,"threads_started":1}
I20260926 13:44:47.216003 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.200547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.1.49.60:43953)
0926 13:44:47.201067 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.201084 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.201144 (+ 60us) server_negotiation.cc:407] Connection header received
0926 13:44:47.201828 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.201845 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.202272 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.202643 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.203716 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.205020 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.206435 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.206939 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.207106 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.211118 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:47.211128 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:47.211145 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:47.211183 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:47.213358 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.213891 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.213897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.213899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.213978 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.214255 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.214258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.214259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.214551 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:47.214772 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:47.215048 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 13:44:47.215210 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":224,"threads_started":1}
I20260926 13:44:47.217206 24064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:44:47 dist-test-slave-1wns krb5kdc[23833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430287, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.60@KRBTEST.COM for kudu/127.1.49.61@KRBTEST.COM
Sep 26 13:44:47 dist-test-slave-1wns krb5kdc[23833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.62@KRBTEST.COM
I20260926 13:44:47.227327 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.217732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.61:35593 (local address 127.0.0.1:46090)
0926 13:44:47.217889 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.217904 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:47.217979 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:47.218269 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:47.218274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:47.218573 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:47.218722 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:47.218733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.219707 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.219715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:47.220407 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:47.220444 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.220962 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.220967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:47.221086 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.221701 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:47.221715 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:47.223355 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:47.225379 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:47.225384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:47.225386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:47.225672 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:47.226071 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:47.226074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:47.226076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:47.226176 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:47.226673 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:47.226680 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:47.226860 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 13:44:47.226951 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 13:44:47.227066 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 13:44:47.227368 23991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.217826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46090 (local address 127.1.49.61:35593)
0926 13:44:47.217947 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.217957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.217969 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:44:47.218075 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.218078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.218129 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.218219 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.218891 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.219548 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.220574 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.220834 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.220922 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.223529 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:47.223532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:47.223534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:47.223561 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:47.225259 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.225797 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.225802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.225805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.225946 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.226276 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.226278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.226280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.226514 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:47.226599 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:47.227048 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 13:44:47.227201 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:44:47.228415 24064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:44:47.229140 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.218569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.1.49.62:40413)
0926 13:44:47.218721 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.218725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.218737 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:44:47.218770 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.218772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.218841 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.218949 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.219761 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.220365 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.221460 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.221816 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.221916 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.224717 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:47.224721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:47.224724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:47.224759 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:47.226633 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.227167 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.227171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.227174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.227270 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.227649 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.227653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.227655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.227969 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:47.228239 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:47.228777 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 13:44:47.228944 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 13:44:47 dist-test-slave-1wns krb5kdc[23833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430287, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.60@KRBTEST.COM for kudu/127.1.49.60@KRBTEST.COM
Sep 26 13:44:47 dist-test-slave-1wns krb5kdc[23833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.49.61@KRBTEST.COM
I20260926 13:44:47.239086 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.228908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45154)
0926 13:44:47.229070 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.229086 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:47.229168 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:47.229771 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:47.229774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:47.229965 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:47.230057 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:47.230064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.231233 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.231236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:47.232014 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:47.232034 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.232581 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.232585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:47.232667 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.233238 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:47.233251 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:47.234978 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:47.237433 (+ 2455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:47.237435 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:47.237437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:47.237649 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:47.237965 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:47.237970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:47.237973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:47.238062 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:47.238481 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:47.238484 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:47.238584 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 13:44:47.238696 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 13:44:47.238871 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 13:44:47.239091 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.229001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.1.49.60:43953)
0926 13:44:47.229120 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.229132 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.229144 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:44:47.229614 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.229617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.229656 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.229761 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.230190 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.231087 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.232171 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.232455 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.232578 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.235127 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:47.235131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:47.235134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:47.235159 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:47.237111 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.237773 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.237777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.237779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.237861 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.238162 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.238167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.238168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.238373 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:47.238484 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:47.238755 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 13:44:47.238969 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":52}
I20260926 13:44:47.240041 23991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.230537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.1.49.61:35593)
0926 13:44:47.230660 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.230664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.230685 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:44:47.230796 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.230798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.230832 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.230907 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.231650 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.232300 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.233278 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.233528 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.233665 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.236362 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:47.236374 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:47.236376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:47.236404 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:47.237961 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.238434 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.238438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.238440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.238533 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.238822 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.238826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.238828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.239114 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:47.239394 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:47.239712 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 13:44:47.239887 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:44:47.243753 24064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3: Bootstrap starting.
I20260926 13:44:47.244216 1220 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:44:47.245630 23888 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "99c165160ad04b9a9591454f1c5ce3d2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560
I20260926 13:44:47.246779 24064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:47.247584 24064 log.cc:824] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:47.250073 24064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3: No bootstrap required, opened a new log
I20260926 13:44:47.253698 24064 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } }
I20260926 13:44:47.253994 24064 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b3a80d23704337962d4c9c5f7917f3, State: Initialized, Role: FOLLOWER
I20260926 13:44:47.254364 24064 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } }
I20260926 13:44:47.254978 24070 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } } }
I20260926 13:44:47.255170 24070 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:44:47.256425 24064 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 13:44:47.262252 24081 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:47.262384 24081 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:47.261274 23996 master_runner.cc:186] Error getting master registration for 127.1.49.62:40413: OK, instance_id { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" instance_seqno: 1790430286687927 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:44:47.265563 23996 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:44:47.333706 23989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } attempt: 4
I20260926 13:44:47.336208 23914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } attempt: 4
Sep 26 13:44:47 dist-test-slave-1wns krb5kdc[23833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430286, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.61@KRBTEST.COM for kudu/127.1.49.60@KRBTEST.COM
Sep 26 13:44:47 dist-test-slave-1wns krb5kdc[23833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430286, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.49.62@KRBTEST.COM for kudu/127.1.49.60@KRBTEST.COM
I20260926 13:44:47.345844 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.334432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45158)
0926 13:44:47.334636 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.334656 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:47.334786 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:47.335175 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:47.335178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:47.335417 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:47.335564 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:47.335574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.336766 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.336773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:47.337729 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:47.337743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.338408 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.338412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:47.338505 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.339148 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:47.339165 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:47.340808 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:47.343156 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:47.343161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:47.343167 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:47.343457 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:47.343912 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:47.343916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:47.343918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:47.344032 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:47.344928 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:47.344931 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:47.345048 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 13:44:47.345465 (+ 417us) client_negotiation.cc:240] Negotiation successful
0926 13:44:47.345632 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 13:44:47.346278 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.334523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.1.49.60:43953)
0926 13:44:47.334720 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.334726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.334749 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:44:47.334956 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.334960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.335022 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.335125 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.335713 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.336607 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.337868 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.338225 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.338773 (+ 548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.340964 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:47.340967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:47.340970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:47.340997 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:47.343006 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.343602 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.343606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.343608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.343692 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.344154 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.344158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.344160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.344483 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:47.344599 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:47.345562 (+ 963us) server_negotiation.cc:299] Negotiation successful
0926 13:44:47.345741 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":402,"server-negotiator.queue_time_us":90}
I20260926 13:44:47.348307 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.336745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.49.60:43953 (local address 127.0.0.1:45160)
0926 13:44:47.336962 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:47.336977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:47.337062 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:47.338338 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:47.338342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:47.338583 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:47.338689 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:47.338695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.340236 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.340241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:47.341115 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:47.341129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.341815 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.341820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:47.341936 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.342439 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:47.342458 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:47.344145 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:47.346446 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:47.346450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:47.346452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:47.346670 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:47.347084 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:47.347087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:47.347089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:47.347196 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:47.347683 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:47.347690 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:47.347848 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 13:44:47.347981 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 13:44:47.348137 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260926 13:44:47.348773 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.336836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45160 (local address 127.1.49.60:43953)
0926 13:44:47.338083 (+ 1247us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.338095 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.338109 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:44:47.338150 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.338154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.338201 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.338298 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.339143 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.340053 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.341267 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.341651 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.342234 (+ 583us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.344306 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:47.344310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:47.344312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:47.344335 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:47.346292 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.346778 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.346780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.346781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.346851 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.347309 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.347313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.347315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.347553 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:47.347668 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:47.348079 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 13:44:47.348210 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1166,"thread_start_us":139,"threads_started":1}
I20260926 13:44:47.349994 23989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef: Bootstrap starting.
I20260926 13:44:47.352218 23914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2: Bootstrap starting.
I20260926 13:44:47.353147 23989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:47.354154 23989 log.cc:824] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:47.355058 23914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:47.356029 23914 log.cc:824] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:47.358565 23914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2: No bootstrap required, opened a new log
I20260926 13:44:47.358709 23989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef: No bootstrap required, opened a new log
I20260926 13:44:47.362141 23914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } }
I20260926 13:44:47.362448 23914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c165160ad04b9a9591454f1c5ce3d2, State: Initialized, Role: FOLLOWER
I20260926 13:44:47.362835 23914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } }
I20260926 13:44:47.363381 23989 raft_consensus.cc:348] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } }
I20260926 13:44:47.363456 24086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } } }
I20260926 13:44:47.363670 24086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:44:47.363686 23989 raft_consensus.cc:729] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd6bc047dc2498884353f8081976bef, State: Initialized, Role: FOLLOWER
I20260926 13:44:47.364135 23989 consensus_queue.cc:260] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } }
I20260926 13:44:47.364485 23914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:47.364569 23888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 13:44:47.364679 23888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:47.365728 23989 sys_catalog.cc:564] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:47.366367 23888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } }
I20260926 13:44:47.367539 23888 leader_election.cc:288] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [CANDIDATE]: Term 1 election: Requested vote from peers efd6bc047dc2498884353f8081976bef (127.1.49.61:35593), b3b3a80d23704337962d4c9c5f7917f3 (127.1.49.60:43953)
I20260926 13:44:47.368338 23963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99c165160ad04b9a9591454f1c5ce3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "efd6bc047dc2498884353f8081976bef"
I20260926 13:44:47.368420 24038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99c165160ad04b9a9591454f1c5ce3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b3b3a80d23704337962d4c9c5f7917f3"
I20260926 13:44:47.368644 23963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:47.368671 24038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:47.369903 23963 raft_consensus.cc:2454] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99c165160ad04b9a9591454f1c5ce3d2 in term 1.
I20260926 13:44:47.370026 24038 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99c165160ad04b9a9591454f1c5ce3d2 in term 1.
I20260926 13:44:47.370523 24087 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } } }
I20260926 13:44:47.370682 24087 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:44:47.371130 23862 leader_election.cc:302] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99c165160ad04b9a9591454f1c5ce3d2, b3b3a80d23704337962d4c9c5f7917f3; no voters:
I20260926 13:44:47.371447 24086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:47.371701 24086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 99c165160ad04b9a9591454f1c5ce3d2, State: Running, Role: LEADER
I20260926 13:44:47.372150 24086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } }
I20260926 13:44:47.374311 24086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99c165160ad04b9a9591454f1c5ce3d2. Latest consensus state: current_term: 1 leader_uuid: "99c165160ad04b9a9591454f1c5ce3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } } }
I20260926 13:44:47.374449 24086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [sys.catalog]: This master's current role is: LEADER
W20260926 13:44:47.376607 24114 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:47.376902 24114 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 13:44:47.388459 24107 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:47.388594 24107 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:47.388672 24118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:47.389773 24118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:47.391255 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.373249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.1.49.60:43953)
0926 13:44:47.373390 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.373395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.380427 (+ 7032us) server_negotiation.cc:407] Connection header received
0926 13:44:47.380639 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.380643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.380694 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.380747 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.381883 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.382783 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.384165 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.384621 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.384790 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.387419 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:47.387424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:47.387427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:47.387461 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:47.389294 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.389881 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.389885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.389887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.389974 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.390253 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.390256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.390258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.390487 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:47.390588 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:47.390957 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 13:44:47.391096 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 13:44:47.401718 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.380330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.1.49.62:40413)
0926 13:44:47.380465 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.380470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.392014 (+ 11544us) server_negotiation.cc:407] Connection header received
0926 13:44:47.392182 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.392186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.392235 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.392305 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.393212 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.394033 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.395341 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.395678 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.395758 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.397929 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:47.397933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:47.397936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:47.397969 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:47.399653 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.400203 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.400207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.400209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.400329 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.400679 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.400682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.400684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.400919 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:47.401032 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:47.401413 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 13:44:47.401553 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:44:47.409519 23991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:47.380886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.1.49.61:35593)
0926 13:44:47.380995 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.380999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.393596 (+ 12597us) server_negotiation.cc:407] Connection header received
0926 13:44:47.393726 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.393729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.393789 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.393884 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.394852 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.395617 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.396840 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:47.397184 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:47.397269 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:47.399404 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:47.399412 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:47.399415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:47.399447 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:47.401221 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.402105 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.402109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.402111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.402209 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:47.408062 (+ 5853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:47.408068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:47.408070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:47.408482 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:47.408640 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:47.409145 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 13:44:47.409327 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:44:47.414647 23963 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef [term 1 FOLLOWER]: Refusing update from remote peer 99c165160ad04b9a9591454f1c5ce3d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:47.414647 24038 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3 [term 1 FOLLOWER]: Refusing update from remote peer 99c165160ad04b9a9591454f1c5ce3d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:47.415762 24086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:47.416395 24111 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:47.420423 24070 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99c165160ad04b9a9591454f1c5ce3d2. Latest consensus state: current_term: 1 leader_uuid: "99c165160ad04b9a9591454f1c5ce3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } } }
I20260926 13:44:47.420432 24087 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99c165160ad04b9a9591454f1c5ce3d2. Latest consensus state: current_term: 1 leader_uuid: "99c165160ad04b9a9591454f1c5ce3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } } }
I20260926 13:44:47.420581 24070 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:44:47.420581 24087 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef [sys.catalog]: This master's current role is: FOLLOWER
W20260926 13:44:47.428457 23942 tablet.cc:2369] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:44:47.433439 23867 tablet.cc:2369] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:44:47.433621 24017 tablet.cc:2369] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:44:47.436685 24070 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99c165160ad04b9a9591454f1c5ce3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } } }
I20260926 13:44:47.436847 24070 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:44:47.437304 24124 mvcc.cc:205] Tried to move back new op lower bound from 7333602457240051712 to 7333602457080373248. Current Snapshot: MvccSnapshot[applied={T|T < 7333602457240051712}]
I20260926 13:44:47.439069 24087 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99c165160ad04b9a9591454f1c5ce3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } } }
I20260926 13:44:47.439214 24087 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:44:47.439378 24086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99c165160ad04b9a9591454f1c5ce3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } } }
I20260926 13:44:47.439500 24086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:47.439608 24086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99c165160ad04b9a9591454f1c5ce3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c165160ad04b9a9591454f1c5ce3d2" member_type: VOTER last_known_addr { host: "127.1.49.62" port: 40413 } } peers { permanent_uuid: "efd6bc047dc2498884353f8081976bef" member_type: VOTER last_known_addr { host: "127.1.49.61" port: 35593 } } peers { permanent_uuid: "b3b3a80d23704337962d4c9c5f7917f3" member_type: VOTER last_known_addr { host: "127.1.49.60" port: 43953 } } }
I20260926 13:44:47.439700 24086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:47.440061 24123 mvcc.cc:205] Tried to move back new op lower bound from 7333602457240051712 to 7333602457080373248. Current Snapshot: MvccSnapshot[applied={T|T < 7333602457240051712}]
I20260926 13:44:47.443436 24118 catalog_manager.cc:1347] Generated new cluster ID: ec1ec8458ecf477bb723e31915677759
I20260926 13:44:47.443516 24118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:47.469583 24118 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:47.471159 24118 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:47.483875 24118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99c165160ad04b9a9591454f1c5ce3d2: Generated new TSK 0
I20260926 13:44:47.484952 24118 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:44:47.516372 24083 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:47.514595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.1.49.60:43953)
0926 13:44:47.514714 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.514718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.515006 (+ 288us) server_negotiation.cc:407] Connection header received
0926 13:44:47.515244 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.515248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.515304 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.515366 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.516208 (+ 842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45174: BlockingRecv error: recv got EOF from 127.0.0.1:45174 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 13:44:47.516639 23917 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:47.514284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.1.49.62:40413)
0926 13:44:47.514423 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.514428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.515571 (+ 1143us) server_negotiation.cc:407] Connection header received
0926 13:44:47.515751 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.515754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.515850 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.515915 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.516510 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60570: BlockingRecv error: recv got EOF from 127.0.0.1:60570 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 13:44:47.517984 23991 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:47.514987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.1.49.61:35593)
0926 13:44:47.515104 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.515108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.516944 (+ 1836us) server_negotiation.cc:407] Connection header received
0926 13:44:47.517140 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.517144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.517203 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.517259 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.517845 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46112: BlockingRecv error: recv got EOF from 127.0.0.1:46112 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 13:44:47.521867 23991 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:47.519886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.1.49.61:35593)
0926 13:44:47.519999 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.520004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.520891 (+ 887us) server_negotiation.cc:407] Connection header received
0926 13:44:47.521069 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.521072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.521158 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.521249 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.521745 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46126: BlockingRecv error: recv got EOF from 127.0.0.1:46126 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 13:44:47.526305 23991 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:47.525262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.1.49.61:35593)
0926 13:44:47.525400 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.525405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.525421 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:44:47.525491 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.525495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.525549 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.525644 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.526184 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46130: BlockingRecv error: recv got EOF from 127.0.0.1:46130 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 13:44:47.531985 23991 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:47.530739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.1.49.61:35593)
0926 13:44:47.530860 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.530865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.531062 (+ 197us) server_negotiation.cc:407] Connection header received
0926 13:44:47.531190 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.531194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.531252 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.531347 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.531866 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46138: BlockingRecv error: recv got EOF from 127.0.0.1:46138 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 13:44:47.539955 23991 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:47.538604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.1.49.61:35593)
0926 13:44:47.538752 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.538759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.538950 (+ 191us) server_negotiation.cc:407] Connection header received
0926 13:44:47.539078 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.539083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.539149 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.539263 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.539759 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46144: BlockingRecv error: recv got EOF from 127.0.0.1:46144 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 13:44:47.552286 23991 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:47.550816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.1.49.61:35593)
0926 13:44:47.550957 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.550962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.551220 (+ 258us) server_negotiation.cc:407] Connection header received
0926 13:44:47.551378 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.551383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.551445 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.551561 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.552152 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46158: BlockingRecv error: recv got EOF from 127.0.0.1:46158 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 13:44:47.572878 23991 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:47.571271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.1.49.61:35593)
0926 13:44:47.571418 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.571425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.571765 (+ 340us) server_negotiation.cc:407] Connection header received
0926 13:44:47.571972 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.571976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.572047 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.572157 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.572735 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46174: BlockingRecv error: recv got EOF from 127.0.0.1:46174 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 13:44:47.608489 23991 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:47.607131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.1.49.61:35593)
0926 13:44:47.607295 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.607300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.607476 (+ 176us) server_negotiation.cc:407] Connection header received
0926 13:44:47.607598 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.607602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.607660 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.607763 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.608360 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46176: BlockingRecv error: recv got EOF from 127.0.0.1:46176 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 13:44:47.676162 23991 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:47.674773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.1.49.61:35593)
0926 13:44:47.674963 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.674968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.675159 (+ 191us) server_negotiation.cc:407] Connection header received
0926 13:44:47.675348 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.675354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.675408 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.675510 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.676039 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46186: BlockingRecv error: recv got EOF from 127.0.0.1:46186 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 13:44:47.808019 23991 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:47.806600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.1.49.61:35593)
0926 13:44:47.806798 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:47.806803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:47.807004 (+ 201us) server_negotiation.cc:407] Connection header received
0926 13:44:47.807075 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:47.807078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:47.807134 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:47.807221 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:47.807823 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46200: BlockingRecv error: recv got EOF from 127.0.0.1:46200 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260926 13:44:48.068290 23991 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:44:48.066841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46216 (local address 127.1.49.61:35593)
0926 13:44:48.067009 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:48.067014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:48.067170 (+ 156us) server_negotiation.cc:407] Connection header received
0926 13:44:48.067351 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:48.067355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:48.067416 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:48.067530 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:48.068128 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46216: BlockingRecv error: recv got EOF from 127.0.0.1:46216 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:44:48.263854 24081 catalog_manager.cc:1259] Loaded cluster ID: ec1ec8458ecf477bb723e31915677759
I20260926 13:44:48.263957 24081 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3: loading cluster ID for follower catalog manager: success
I20260926 13:44:48.266369 24081 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3: acquiring CA information for follower catalog manager: success
I20260926 13:44:48.267962 24081 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3b3a80d23704337962d4c9c5f7917f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:44:48.378373 24114 catalog_manager.cc:1259] Loaded cluster ID: ec1ec8458ecf477bb723e31915677759
I20260926 13:44:48.378571 24114 catalog_manager.cc:1552] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef: loading cluster ID for follower catalog manager: success
I20260926 13:44:48.380541 24114 catalog_manager.cc:1574] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef: acquiring CA information for follower catalog manager: success
I20260926 13:44:48.381916 24114 catalog_manager.cc:1602] T 00000000000000000000000000000000 P efd6bc047dc2498884353f8081976bef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:44:48.587481 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:48.583050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.1.49.61:35593)
0926 13:44:48.583350 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:48.583354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:48.583465 (+ 111us) server_negotiation.cc:407] Connection header received
0926 13:44:48.583573 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:48.583576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:48.583745 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:48.583937 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:48.584221 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:48.584873 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:48.585751 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:48.586015 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:48.586105 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:48.586500 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:48.586990 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:48.587092 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:48.587190 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 13:44:48.587248 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":98,"threads_started":1}
I20260926 13:44:48.595054 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:48.591082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45176 (local address 127.1.49.60:43953)
0926 13:44:48.591320 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:48.591325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:48.591341 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:44:48.591459 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:48.591462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:48.591583 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:48.591718 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:48.591988 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:48.592520 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:48.593417 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:48.593670 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:48.593785 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:48.594117 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:48.594594 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:48.594738 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:48.594804 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 13:44:48.594874 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260926 13:44:48.603247 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:48.598897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.1.49.61:35593)
0926 13:44:48.599036 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:48.599041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:48.599310 (+ 269us) server_negotiation.cc:407] Connection header received
0926 13:44:48.599422 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:48.599426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:48.599489 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:48.599607 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:48.599921 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:48.600748 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:48.601927 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:48.602254 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:48.602345 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:48.602763 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:48.602838 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:48.602895 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:48.603017 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 13:44:48.603068 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:44:48.603865 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:48.598727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.1.49.62:40413)
0926 13:44:48.599149 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:48.599154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:48.599167 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:44:48.599281 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:48.599284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:48.599468 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:48.599665 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:48.600148 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:48.600931 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:48.601867 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:48.602221 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:48.602336 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:48.602748 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:48.603140 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:48.603232 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:48.603603 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 13:44:48.603673 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":78,"threads_started":1}
I20260926 13:44:48.603971 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:48.599046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.1.49.60:43953)
0926 13:44:48.599183 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:48.599188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:48.599423 (+ 235us) server_negotiation.cc:407] Connection header received
0926 13:44:48.599602 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:48.599608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:48.599665 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:48.599735 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:48.599985 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:48.600639 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:48.602046 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:48.602296 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:48.602397 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:48.603459 (+ 1062us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:48.603548 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:48.603597 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:48.603761 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 13:44:48.603839 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 13:44:48.613992 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 23846
I20260926 13:44:48.620963 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 23921
I20260926 13:44:48.627292 1220 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7nlTrX/build/debug/bin/kudu with pid 23996
2026-09-26T13:44:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3377 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6750 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (304780 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST